我有两个以逗号分隔的字符串列(sourceAuthors
and targetAuthors
).
val df = Seq(
("Author1,Author2,Author3","Author2,Author3,Author1")
).toDF("source","target")
我想添加另一列nCommonAuthors
与共同作者的数量。
我尝试通过这种方式做到这一点:
def myUDF = udf { (s1: String, s2: String) =>
s1.split(",")
s2.split(",")
s1.intersect(s2).length
}
val newDF = myDF.withColumn("nCommonAuthors", myUDF($"source", $"target"))
我收到以下错误:
线程“main”中的异常 java.lang.UnsupportedOperationException:不支持类型 Unit 的架构
知道为什么我会收到此错误吗?如何找到两列之间的共同元素?
除非我误解了你的问题,否则有一些标准函数可以帮助你(这样你就不必编写 UDF),即split
and array_intersect
.
给定以下数据集:
val df = Seq(("Author1,Author2,Author3","Author2,Author3"))
.toDF("source","target")
scala> df.show(false)
+-----------------------+---------------+
|source |target |
+-----------------------+---------------+
|Author1,Author2,Author3|Author2,Author3|
+-----------------------+---------------+
您可以编写以下结构化查询:
val intersect = array_intersect(split('source, ","), split('target, ","))
val solution = df.select(intersect as "common_elements")
scala> solution.show(false)
+------------------+
|common_elements |
+------------------+
|[Author2, Author3]|
+------------------+
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)