假设我有以下数据框：
val df1 = sc.parallelize(Seq("a1" -> "a2", "b1" -> "b2", "c1" -> "c2")).toDF("a", "b")
val df2 = sc.parallelize(Seq("aa1" -> "aa2", "bb1" -> "bb2")).toDF("aa", "bb")
我想要以下内容：
| a | b | aa | bb |
----------------------
| a1 | a2 | aa1 | aa2 |
| a1 | a2 | bb1 | bb2 |
| b1 | b2 | aa1 | aa2 |
| b1 | b2 | bb1 | bb2 |
| c1 | c2 | aa1 | aa2 |
| c1 | c2 | bb1 | bb2 |
所以每一行df1映射到所有行df2。我这样做的方式如下：
val df1_dummy = df1.withColumn("dummy_df1", lit("dummy"))
val df2_dummy = df2.withColumn("dummy_df2", lit("dummy"))
val desired_result = df1_dummy
.join(df2_dummy, $"dummy_df1" === $"dummy_df2", "left")
.drop("dummy_df1")
.drop("dummy_df2")
它给出了期望的结果，但似乎有点糟糕。有更有效的方法吗？有什么推荐吗？
就是这样crossJoin is for:
val result = df1.crossJoin(df2)
result.show()
// +---+---+---+---+
// |a |b |aa |bb |
// +---+---+---+---+
// |a1 |a2 |aa1|aa2|
// |a1 |a2 |bb1|bb2|
// |b1 |b2 |aa1|aa2|
// |b1 |b2 |bb1|bb2|
// |c1 |c2 |aa1|aa2|
// |c1 |c2 |bb1|bb2|
// +---+---+---+---+
本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)
