为了完整起见,这里是一个使用的解决方案scipy.stats.pearsonr
(docs https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.pearsonr.html) 创建 p 值矩阵。创建布尔掩码以传递给seaborn(或另外与numpy结合使用)之后np.triu
隐藏相关性的上三角)
def corr_sig(df=None):
p_matrix = np.zeros(shape=(df.shape[1],df.shape[1]))
for col in df.columns:
for col2 in df.drop(col,axis=1).columns:
_ , p = stats.pearsonr(df[col],df[col2])
p_matrix[df.columns.to_list().index(col),df.columns.to_list().index(col2)] = p
return p_matrix
p_values = corr_sig(df)
mask = np.invert(np.tril(p_values<0.05))
# note seaborn will hide correlation were the boolean value is True in the mask
完整流程及示例
首先创建一些样本数据(3 个相关变量;3 个不相关变量):
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
# Simulate 3 correlated variables
num_samples = 100
mu = np.array([5.0, 0.0, 10.0])
# The desired covariance matrix.
r = np.array([
[ 3.40, -2.75, -2.00],
[ -2.75, 5.50, 1.50],
[ -2.00, 1.50, 1.25]
])
y = np.random.multivariate_normal(mu, r, size=num_samples)
df = pd.DataFrame(y)
df.columns = ["Correlated1","Correlated2","Correlated3"]
# Create two random variables
for i in range(2):
df.loc[:,f"Uncorrelated{i}"] = np.random.randint(-2000,2000,len(df))
# To make sure that they are uncorrelated - add also a nearly invariant variables
df.loc[:,"Near Invariant"] = np.random.randint(-99,-95,num_samples)
绘图功能,方便使用
主要用于热图的修饰。
def plot_cor_matrix(corr, mask=None):
f, ax = plt.subplots(figsize=(11, 9))
sns.heatmap(corr, ax=ax,
mask=mask,
# cosmetics
annot=True, vmin=-1, vmax=1, center=0,
cmap='coolwarm', linewidths=2, linecolor='black', cbar_kws={'orientation': 'horizontal'})
具有所有相关性的示例数据的校正图
为了让您了解此示例相关矩阵中的相关性如何不过滤显着相关性(p 值
# Plotting without significance filtering
corr = df.corr()
mask = np.triu(corr)
plot_cor_matrix(corr,mask)
plt.show()
仅包含 Sig 的示例数据的 Corr.Plot。相关性最后绘制仅具有显着 p 值相关性的图 (alpha
# Plotting with significance filter
corr = df.corr() # get correlation
p_values = corr_sig(df) # get p-Value
mask = np.invert(np.tril(p_values<0.05)) # mask - only get significant corr
plot_cor_matrix(corr,mask)
结论
而在第一个相关矩阵中有一些相关系数(r
) 大于 .05 (按照OP评论中的建议进行过滤),这并不意味着 p 值显着。因此,区分p
相关系数的值r
.
我希望这个答案将来能够帮助其他人寻找一种方法来绘制与a的显着相关性sns.heatmap