scipy 模块scipy.spatial.distance http://docs.scipy.org/doc/scipy/reference/spatial.distance.html包括称为距离函数皮尔逊距离 https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Pearson.E2.80.99s_distance,即 1 减去相关系数。通过使用参数metric='correlation'
in scipy.spatial.distance.cdist http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html,您可以有效计算两个输入中每对向量的 Pearson 相关系数。
这是一个例子。我将修改您的数据,使系数更加多样化:
In [96]: list1 = [[1, 2, 3.5], [4, 5, 6], [7, 8, 12], [10, 7, 10]]
In [97]: list2 = [[10, 20, 30], [41, 51, 60], [77, 80, 79], [80, 78, 56]]
所以我们知道会发生什么,这里是使用计算得出的相关系数scipy.stats.pearsonr
:
In [98]: [pearsonr(x, y)[0] for x in list1 for y in list2]
Out[98]:
[0.99339926779878296,
0.98945694873927104,
0.56362148019067804,
-0.94491118252306794,
1.0,
0.99953863896044937,
0.65465367070797709,
-0.90112711377916588,
0.94491118252306805,
0.93453339271427294,
0.37115374447904509,
-0.99339926779878274,
0.0,
-0.030372836961539348,
-0.7559289460184544,
-0.43355498476205995]
在数组中查看它们更方便:
In [99]: np.array([pearsonr(x, y)[0] for x in list1 for y in list2]).reshape(len(list1), len(list2))
Out[99]:
array([[ 0.99339927, 0.98945695, 0.56362148, -0.94491118],
[ 1. , 0.99953864, 0.65465367, -0.90112711],
[ 0.94491118, 0.93453339, 0.37115374, -0.99339927],
[ 0. , -0.03037284, -0.75592895, -0.43355498]])
这是使用计算得出的相同结果cdist
:
In [100]: from scipy.spatial.distance import cdist
In [101]: 1 - cdist(list1, list2, metric='correlation')
Out[101]:
array([[ 0.99339927, 0.98945695, 0.56362148, -0.94491118],
[ 1. , 0.99953864, 0.65465367, -0.90112711],
[ 0.94491118, 0.93453339, 0.37115374, -0.99339927],
[ 0. , -0.03037284, -0.75592895, -0.43355498]])
Using cdist
is much比打电话更快pearsonr
在嵌套循环中。这里我将使用两个数组,data1
and data2
,每个大小为 (100, 10000):
In [102]: data1 = np.random.randn(100, 10000)
In [103]: data2 = np.random.randn(100, 10000)
我会用方便的%timeit
命令输入ipython
测量执行时间:
In [104]: %timeit c1 = [pearsonr(x, y)[0] for x in data1 for y in data2]
1 loop, best of 3: 836 ms per loop
In [105]: %timeit c2 = 1 - cdist(data1, data2, metric='correlation')
100 loops, best of 3: 4.35 ms per loop
嵌套循环需要 836 毫秒,cdist
.