从 DataFrame 中删除强相关列[重复]

我有一个像这样的数据框

dict_ = {'Date':['2018-01-01','2018-01-02','2018-01-03','2018-01-04','2018-01-05'],'Col1':[1,2,3,4,5],'Col2':[1.1,1.2,1.3,1.4,1.5],'Col3':[0.33,0.98,1.54,0.01,0.99]}
df = pd.DataFrame(dict_, columns=dict_.keys())

然后，我计算列之间的皮尔逊相关性，并过滤掉相关性高于阈值 0.95 的列

def trimm_correlated(df_in, threshold):
    df_corr = df_in.corr(method='pearson', min_periods=1)
    df_not_correlated = ~(df_corr.mask(np.eye(len(df_corr), dtype=bool)).abs() > threshold).any()
    un_corr_idx = df_not_correlated.loc[df_not_correlated[df_not_correlated.index] == True].index
    df_out = df_in[un_corr_idx]
    return df_out

这产生

uncorrelated_factors = trimm_correlated(df, 0.95)
print uncorrelated_factors

    Col3
0   0.33
1   0.98
2   1.54
3   0.01
4   0.99

到目前为止，我对结果感到满意，但我想从每个相关对中保留一列，因此在上面的示例中我想包含 Col1 或 Col2。为了得到某物。像这样

    Col1   Col3
0    1     0.33
1    2     0.98
2    3     1.54
3    4     0.01
4    5     0.99

另外，我可以做进一步的评估来确定保留哪些相关列吗？

thanks

您可以使用np.tril() https://docs.scipy.org/doc/numpy/reference/generated/numpy.tril.html#numpy.tril代替np.eye()对于面具：

def trimm_correlated(df_in, threshold):
    df_corr = df_in.corr(method='pearson', min_periods=1)
    df_not_correlated = ~(df_corr.mask(np.tril(np.ones([len(df_corr)]*2, dtype=bool))).abs() > threshold).any()
    un_corr_idx = df_not_correlated.loc[df_not_correlated[df_not_correlated.index] == True].index
    df_out = df_in[un_corr_idx]
    return df_out

Output:

    Col1    Col3
0   1       0.33
1   2       0.98
2   3       1.54
3   4       0.01
4   5       0.99