这个解决方案应该可行。这里是实现 hashCode 和 equals 的案例类(我们可以将它们称为比较器)。
可以根据不同的密文修改/更新hashCode和equals
case class Year(var year:Int){
override def hashCode(): Int = {
this.year = this.year match {
case 2012 => 2012
case 12 => 2012
case 13 => 2013
case _ => this.year
}
this.year.hashCode()
}
override def equals(that: Any): Boolean ={
val year1 = 2000 + that.asInstanceOf[Year].year % 100
val year2 = 2000 + this.year % 100
if (year1 == year2)
true
else
false
}
}
case class Month(var month:String){
override def hashCode(): Int = {
this.month = this.month match {
case "January" => "Jan"
case "Janu" => "Jan"
case "February" => "Feb"
case "Febr" => "Feb"
case _ => this.month
}
this.month.hashCode
}
override def equals(that: Any): Boolean ={
val month1 = this.month match {
case "January" => "Jan"
case "Janu" => "Jan"
case "February" => "Feb"
case "Febr" => "Feb"
case _ => this.month
}
val month2 = that.asInstanceOf[Month].month match {
case "January" => "Jan"
case "Janu" => "Jan"
case "February" => "Feb"
case "Febr" => "Feb"
case _ => that.asInstanceOf[Month].month
}
if (month1.equals(month2))
true
else
false
}
}
这是分组键的重要比较器,它仅使用单独的 col 比较器
case class Key(var year:Year, var month:Month){
override def hashCode(): Int ={
this.year.hashCode() + this.month.hashCode()
}
override def equals(that: Any): Boolean ={
if ( this.year.equals(that.asInstanceOf[Key].year) && this.month.equals(that.asInstanceOf[Key].month))
true
else
false
}
}
case class Record(year:Int,month:String,counter:Int)
val df = spark.read.format("com.databricks.spark.csv")
.option("header", "true")
.option("inferSchema", "true")
.load("data.csv").as[Record]
df.rdd.groupBy[Key](
(record:Record)=>Key(Year(record.year), Month(record.month)))
.map(x=> Record(x._1.year.year, x._1.month.month, x._2.toList.map(_.counter).sum))
.toDS().show()
这使
+----+-----+-------+
|year|month|counter|
+----+-----+-------+
|2012| Feb| 800|
|2013| Feb| 500|
|2012| Jan| 700|
+----+-----+-------+
for this input in data.csv
Year,Month,Counter
2012,February,400
2012,Jan,100
12,January,200
12,Janu,300
2012,Feb,400
13,Febr,500
2012,Jan,100
请注意,对于案例类“年”和“月”,还将值更新为标准值(否则无法预测它选择哪个值)。