我在运行 MapReduce WordCount 作业时遇到一些错误。
错误:java.io.IOException:所有收集器的初始化
失败的。最后一个收集器中的错误是:class wordcount.wordmapper at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:414)
在 org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81) 在
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 在
java.security.AccessController.doPrivileged(本机方法)位于
javax.security.auth.Subject.doAs(Subject.java:415)
atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 原因:
java.lang.ClassCastException: 类 wordcount.wordmapperat
java.lang.Class.asSubclass(Class.java:3165)at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:892)
在
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1005)
在
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
我发现我必须实施WritableComparable<>
使其工作的接口。
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
public class Pair implements WritableComparable<Pair> {
public Text key1;
public Text key2;
public Pair() {
key1 = new Text();
key2 = new Text();
}
public Pair(String key1, String key2) {
this.key1 = new Text(key1);
this.key2 = new Text(key2);
}
public void setKey1(Text key1) {
this.key1 = key1;
}
public void setKey2(Text key2) {
this.key2 = key2;
}
public Text getKey1() {
return key1;
}
public Text getKey2() {
return key2;
}
@Override
public boolean equals(Object b) {
Pair p = (Pair) b;
return p.key1.toString().equals(this.key1.toString())
&& p.key2.toString().equals(this.key2.toString());
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result
+ ((key1 == null) ? 0 : key1.toString().hashCode());
result = prime * result
+ ((key2 == null) ? 0 : key2.toString().hashCode());
return result;
}
@Override
public String toString() {
return "(" + key1 + ", " + key2 + ")";
}
@Override
public void readFields(DataInput arg0) throws IOException {
key1.readFields(arg0);
key2.readFields(arg0);
}
@Override
public void write(DataOutput arg0) throws IOException {
key1.write(arg0);
key2.write(arg0);
}
public int compareTo(Pair p1) {
int k = this.key1.toString().compareTo(p1.key1.toString());
if (k != 0) {
return k;
} else {
return this.key2.toString().compareTo(p1.key2.toString());
}
}
}
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)