我通过 SSH 连接到 AWS EMR v5.4.0 实例，并且想要调用 s3distcp。这个链接 http://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html演示了如何设置 emr 步骤来调用它，但是当我运行它时，出现以下错误：
Container launch failed for container_1492469375740_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:390)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
我跟着此处的说明 https://stackoverflow.com/questions/26381540/the-auxservicemapreduce-shuffle-does-not-exist但还是没用。
事实证明我需要重启yarn节点管理器服务 https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/配置后mapreduce_shuffle:
$ initctl list | grep yarn
hadoop-yarn-resourcemanager start/running, process 1256
hadoop-yarn-proxyserver start/running, process 702
hadoop-yarn-nodemanager start/running, process 896
$ sudo stop hadoop-yarn-nodemanager
$ sudo start hadoop-yarn-nodemanager
另外，如果它有助于yarn-site.xml文件位于：/etc/hadoop/conf/yarn-site.xml。它已经有一个条目yarn.nodemanager.aux-services但mapreduce_shuffle没有配置：
<property>
<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
所以我这样添加：
<property>
<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)
