join
是的成员org.apache.spark.rdd.PairRDDFunctions
。那么为什么隐式类没有触发呢?
scala> val s = Seq[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }]()
scala> val r = sc.parallelize(s)
scala> r.join(r) // Gives your error message.
scala> val p = new org.apache.spark.rdd.PairRDDFunctions(r)
<console>:25: error: no type parameters for constructor PairRDDFunctions: (self: org.apache.spark.rdd.RDD[(K, V)])(implicit kt: scala.reflect.ClassTag[K], implicit vt: scala.reflect.ClassTag[V], implicit ord: Ordering[K])org.apache.spark.rdd.PairRDDFunctions[K,V] exist so that it can be applied to arguments (org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }])
--- because ---
argument expression's type is not compatible with formal parameter type;
found : org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }]
required: org.apache.spark.rdd.RDD[(?K, ?V)]
Note: (Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) } >: (?K, ?V), but class RDD is invariant in type T.
You may wish to define T as -T instead. (SLS 4.5)
val p = new org.apache.spark.rdd.PairRDDFunctions(r)
^
<console>:25: error: type mismatch;
found : org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }]
required: org.apache.spark.rdd.RDD[(K, V)]
val p = new org.apache.spark.rdd.PairRDDFunctions(r)
我确信其他人都清楚该错误消息,但对于我自己来说,让我们尝试理解它。PairRDDFunctions
有两个类型参数,K
and V
. Your forSome
是针对整个对的,因此不能将其分成单独的K
and V
类型。没有K
and V
that RDD[(K, V)]
等于你的 RDD 类型。
但是,您可以拥有forSome
仅适用于密钥,而不适用于整个密钥对。 Join 现在可以工作了,因为这种类型可以分为K
and V
.
scala> val s2 = Seq[(Long, (Int, (Long, String, Array[_0])) forSome { type _0 <: (String, Double) })]()
scala> val r2 = sc.parallelize(2s)
scala> r2.join(r2)
res0: org.apache.spark.rdd.RDD[(Long, ((Int, (Long, String, Array[_0])) forSome { type _0 <: (String, Double) }, (Int, (Long, String, Array[_0])) forSome { type _0 <: (String, Double) }))] = MapPartitionsRDD[5] at join at <console>:26