我用它来删除重复的行
public class DLines
{
public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable>
{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
@Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException
{
String line=value.toString();
//int hash_code=line.hashCode();
context.write(value, one);
}
}
public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, IntWritable>
{
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException
{
public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException
{
int sum = 0;
for (IntWritable value : values)
{
sum += value.get();
}
if (sum<2)
{
context.write(key,new IntWritable(sum));
}
}
}
我必须只将密钥存储在 hdfs 中。
如果您不需要减速器的价值,只需使用空可写.
你可以简单地说context.write(key,NullWritable.get());
在你的驱动程序中,你还可以设置
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
&
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)