[ Spark ] java.lang.NoSuchMethodError: com.google.common.hash.Funnels.stringFunnel

问题

• 相关环境参数

spark.version：2.1.0
scala.version：2.11.8
guava：22.0

• 代码中使用到 guava 的布隆过滤器，产生如下报错

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.hash.Funnels.stringFunnel(Ljava/nio/charset/Charset;)Lcom/google/common/hash/Funnel;
	at BloomFilterCase$.main(BloomFilterCase.scala:11)
	at BloomFilterCase.main(BloomFilterCase.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决

• spark-core_2.11 中也有使用到 guava，先排除掉

<exclusions>
    <exclusion>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
    </exclusion>
</exclusions>

两种方式

一. 初始化 spark 时，设置优先使用 jar 包中的依赖

val spark = SparkSession.builder
      .config("spark.executor.userClassPathFirst", value = true)
      .config("spark.driver.userClassPathFirst", value = true)
      .getOrCreate()

• 这种方式可以解决 guava 的报错，但是会带来新的问题
• 如下 序列化的问题

Caused by: java.io.InvalidClassException: org.apache.spark.scheduler.Task;
 local class incompatible: stream classdesc serialVersionUID = 
 -3444186963810984713, local class serialVersionUID = -8136233304550334863

二. 使用 shade 插件

• 需要去掉 assembly 插件

maven-assembly-plugin

• 引入 maven-shade-plugin 插件

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <!-- 设置主类  -->
                        <mainClass>MXHistoryBloomFilter</mainClass>
                    </transformer>
                </transformers>
                <relocations>
                    <relocation>
                        <pattern>com.google.common</pattern>
                        <shadedPattern>my_guava.common</shadedPattern>
                    </relocation>
                </relocations>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/maven/**</exclude>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>

参考链接

• https://zhuanlan.zhihu.com/p/44956574
• https://blog.csdn.net/adorechen/article/details/90722933