我正在使用 Spark Scala 来计算 Dataframe 行之间的余弦相似度。
数据帧格式如下
root
|-- SKU: double (nullable = true)
|-- Features: vector (nullable = true)
下面的数据框示例
+-------+--------------------+
| SKU| Features|
+-------+--------------------+
| 9970.0|[4.7143,0.0,5.785...|
|19676.0|[5.5,0.0,6.4286,4...|
| 3296.0|[4.7143,1.4286,6....|
|13658.0|[6.2857,0.7143,4....|
| 1.0|[4.2308,0.7692,5....|
| 513.0|[3.0,0.0,4.9091,5...|
| 3753.0|[5.9231,0.0,4.846...|
|14967.0|[4.5833,0.8333,5....|
| 2803.0|[4.2308,0.0,4.846...|
|11879.0|[3.1429,0.0,4.5,4...|
+-------+--------------------+
我尝试转置矩阵并检查以下提到的链接。Apache Spark Python DataFrame 上的余弦相似度 https://stackoverflow.com/questions/43921636/apache-spark-python-cosine-similarity-over-dataframes, 使用 tf-idf 将文本特征化为向量来计算余弦相似度 https://stackoverflow.com/questions/32645231/calculating-cosine-similarity-by-featurizing-the-text-into-vector-using-tf-idf但我相信有更好的解决方案
我尝试了下面的示例代码
val irm = new IndexedRowMatrix(inClusters.rdd.map {
case (v,i:Vector) => IndexedRow(v, i)
}).toCoordinateMatrix.transpose.toRowMatrix.columnSimilarities
但我收到以下错误
Error:(80, 12) constructor cannot be instantiated to expected type;
found : (T1, T2)
required: org.apache.spark.sql.Row
case (v,i:Vector) => IndexedRow(v, i)
我检查了以下链接Apache Spark:如何从 DataFrame 创建矩阵? https://stackoverflow.com/questions/31567989/apache-spark-how-to-create-a-matrix-from-a-dataframe但无法使用 Scala 做到这一点