如果您必须使用reduceByKey函数,请尝试以下解决方案:
SCALA:
val df = sc.parallelize(Seq(("Mumbai", 19, 30),
("Delhi", 5, 41),
("Kolkata", 20, 40),
("Mumbai", 18, 35),
("Delhi", 4, 42),
("Delhi", 10, 44),
("Kolkata", 19, 39))).map(x => (x._1,x._2)).keyBy(_._1)
df.reduceByKey((accum, n) => if (accum._2 > n._2) n else accum).map(_._2).collect().foreach(println)
PYTHON:
rdd = sc.parallelize([("Mumbai", 19, 30),
("Delhi", 5, 41),
("Kolkata", 20, 40),
("Mumbai", 18, 35),
("Delhi", 4, 42),
("Delhi", 10, 44),
("Kolkata", 19, 39)])
def reduceFunc(accum, n):
print(accum, n)
if accum[1] > n[1]:
return(n)
else: return(accum)
def mapFunc(lines):
return (lines[0], lines[1])
rdd.map(mapFunc).keyBy(lambda x: x[0]).reduceByKey(reduceFunc).map(lambda x : x[1]).collect()
Output:
(Kolkata,19)
(Delhi,4)
(Mumbai,18)
如果你不想做一个reduceByKey。只需一组后跟 min 函数即可得到所需的结果。
val df = sc.parallelize(Seq(("Mumbai", 19, 30),
("Delhi", 5, 41),
("Kolkata", 20, 40),
("Mumbai", 18, 35),
("Delhi", 4, 42),
("Delhi", 10, 44),
("Kolkata", 19, 39))).toDF("city", "minTemp", "maxTemp")
df.groupBy("city").agg(min("minTemp")).show
Output :
+-------+------------+
| city|min(minTemp)|
+-------+------------+
| Mumbai| 18|
|Kolkata| 19|
| Delhi| 4|
+-------+------------+