运行scala代码jar出现NoSuchMethodError:scala.Predef$.refArrayOps

我的代码在idea中通过本地模式可以正常运行，当我打印成jar包上传到我部署的SPARK服务器上运行时，出现NoSuchMethodError: scala.预定义 $. refArrayOps 出现了。 出错的那行代码如下val expectArray=expectVertex.take(2).toArray.sortBy（it=>｛it_1｝）ExpectVertex是一个scala映射，它的键类型是graphx.VertexId，它的值类型是Int

我在使用Spark简单代码执行此操作时也遇到了这个问题，当我使用数组函数的一行时出现此错误，代码如下 包 org.example

import org.apache.spark.graphx.{Edge, Graph}
import org.apache.spark.{SparkConf, SparkContext}

import java.util.logging.{Level, Logger}

/**
 * Hello world!
 *
 */
class App{
  def run(): Unit ={
    Logger.getLogger("org.apache.spark").setLevel(Level.WARNING)
    Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF)
    val conf = new SparkConf().setAppName("AXU test")
      .setMaster("local")
    val sc = new SparkContext(conf)
    val vertices = sc.parallelize(Array((1L, "A"), (2L, "B"), (3L, "C"), (4L, "D")))
    val edges = sc.parallelize(Array(Edge(1L, 2L, "friend"), Edge(2L, 3L, "follow"), Edge(3L, 4L, "friend")))
    val graph = Graph(vertices, edges)
    val inDegrees = graph.inDegrees
    inDegrees.collect().foreach(println)
    val deg = inDegrees.collect()
    for( i <- 0 to deg.length-1){
      print("this is no." + (i+1) + " point indegree:")
      println("id: " + deg(i)._1 + " value: " + deg(i)._2)
    }
    sc.stop()
  }
}

日志是

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
    at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
    at org.example.App.run(App.scala:23)
    at org.example.Main$.main(Main.scala:6)
    at org.example.Main.main(Main.scala)

如果我删除第23行中的代码，则代码是inDegrees.collect().foreach(println)它可以正常工作。 我编译和运行的scala版本都是2.12.7。 看来我不能使用像Array [T]这样的方法。 foreach 或数组 [T]。 jar包中的sortBy（it=>{it_1}）（我使用Maven打包jar）。 maven内容如下。

    <properties>
        <scala.version>2.12.7</scala.version>
        <spark.version>2.4.4</spark.version>
    </properties>


    <build>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <id>compile-scala</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>test-compile-scala</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>org.example.Main</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>assembly</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.6.0</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <executable>java</executable>
                    <includeProjectDependencies>true</includeProjectDependencies>
                    <includePluginDependencies>false</includePluginDependencies>
                    <classpathScope>compile</classpathScope>
                    <mainClass>org.example.Main</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

有人能告诉我为什么会出现这个问题吗？先感谢您。

您很可能使用 Scala 2.12 在本地编译代码，但在服务器上它使用 Scala 2.13 或 2.11 运行。

尝试使用服务器上的 Scala 版本重新编译代码。

Scala 2.11、2.12、2.13 二进制不兼容。

的签名refArrayOps不同（以二进制不兼容的方式）

• 在 Scala 2.13 中

def refArrayOps(scala.Array[scala.Any]): scala.Any（头皮）

public <T> T[] refArrayOps(T[])（javap）scalap 和 javap 显示不同的方法签名 https://stackoverflow.com/questions/44121009/scalap-and-javap-showing-different-method-signature

@inline implicit def refArrayOps[T <: AnyRef](xs: Array[T]): ArrayOps[T] [api https://www.scala-lang.org/api/2.13.10/scala/Predef%24.html#refArrayOps%5BT%3C:AnyRef%5D(xs:Array%5BT%5D):scala.collection.ArrayOps%5BT%5D] [sources https://github.com/scala/scala/blob/v2.13.10/src/library/scala/Predef.scala#L464]

• 在 Scala 2.12 中

def refArrayOps(scala.Array[scala.Any]): scala.Array[scala.Any]（头皮）

public <T> T[] refArrayOps(T[]) (javap)

implicit def refArrayOps[T <: AnyRef](xs: Array[T]): ArrayOps.ofRef[T] [api https://www.scala-lang.org/api/2.12.17/scala/Predef%24.html#refArrayOps%5BT%3C:AnyRef%5D(xs:Array%5BT%5D):scala.collection.mutable.ArrayOps.ofRef%5BT%5D] [sources https://github.com/scala/scala/blob/v2.12.17/src/library/scala/Predef.scala#L450]

• 在 Scala 2.11-

def refArrayOps(scala.Array[scala.Any]): scala.collection.mutable.ArrayOps（头皮）

public <T> scala.collection.mutable.ArrayOps<T> refArrayOps(T[]) (javap)

implicit def refArrayOps[T <: AnyRef](xs: Array[T]): ArrayOps[T] [api 2.11 https://www.scala-lang.org/api/2.11.12/index.html#scala.Predef%24@refArrayOps%5BT%3C:AnyRef%5D(xs:Array%5BT%5D):scala.collection.mutable.ArrayOps%5BT%5D 2.10 https://www.scala-lang.org/api/2.10.7/index.html#scala.Predef%24] [来源2.11 https://github.com/scala/scala/blob/v2.11.12/src/library/scala/Predef.scala#L344 2.10 https://github.com/scala/scala/blob/v2.10.7/src/library/scala/Predef.scala#L334 2.9 https://github.com/scala/scala/blob/v2.9.3/src/library/scala/Predef.scala#L295]

Kafka 在 MAC 上启动错误 .. 与 java 和 scala 相关的东西 ... NoSuchMethodError: scala.Predef$.refArrayOps https://stackoverflow.com/questions/75712355/kafka-start-error-on-mac-something-related-to-java-and-scala-nosuchmethod

java.lang.NoSuchMethodError: scala.Predef$.refArrayOps https://stackoverflow.com/questions/40328948/java-lang-nosuchmethoderror-scala-predef-refarrayops

如何修复 NoSuchMethodError？ https://stackoverflow.com/questions/35186/how-do-i-fix-a-nosuchmethoderror

尝试将表保存到 Hive 时出现 java.lang.NoSuchMethodError：org.apache.hadoop.hive.common.FileUtils.mkdir https://stackoverflow.com/questions/75880410/java-lang-nosuchmethoderror-org-apache-hadoop-hive-common-fileutils-mkdir-while

java.lang.NoSuchMethodError：Scala Spark 作业中的 scala.Predef$.refArrayOps https://stackoverflow.com/questions/61675153/java-lang-nosuchmethoderror-scala-predef-refarrayops-in-spark-job-with-scala

你可以运行

import java.net.URLClassLoader
import java.util.Arrays

//  List.unfold(getClass.getClassLoader) { cl =>
//    val urls = s"classloader: ${cl.getClass.getName}" :: 
//      (cl match {
//        case cl: URLClassLoader =>
//          "classloader urls:" :: 
//            cl.getURLs.map(_.toString).toList
//        case _ => List("not URLClassLoader")
//      })
//    Option.when(cl != null)((urls, cl.getParent))
//  }.flatten.foreach(println)

var cl = getClass.getClassLoader
while (cl != null) {
  println(s"classloader: ${cl.getClass.getName}")
  cl match {
    case cl: URLClassLoader =>
      println("classloader urls:")
      // cl.getURLs.foreach(println) // uses Scala refArrayOps again
      println(Arrays.toString(cl.getURLs.asInstanceOf[Array[Object]])) // pure Java
    case _ =>
      println("not URLClassLoader")
  }
  cl = cl.getParent
}

or

println(
  System.getProperty("java.class.path")
)

(System.getProperty("java.class.path") 和 getClassLoader.getURLs() 有什么区别？ https://stackoverflow.com/questions/18626396/whats-the-difference-between-system-getpropertyjava-class-path-and-getclass)

在您正在使用的实际 Spark 环境中。然后你会看到你的类路径。是否有不同scala-library, 是否有不同_2.11, _2.12, _2.13依赖关系。

https://www.scala-sbt.org/1.x/docs/Howto-Classpaths.html https://www.scala-sbt.org/1.x/docs/Howto-Classpaths.html

scalacOptions += "-Ylog-classpath"

scala -version显示系统中安装的 Scala。类路径中的 Scala 可能不同。

为什么 SBT 显示的 ScalaVersion 与我的系统不同？ https://stackoverflow.com/questions/75531849/why-does-sbt-show-a-different-scalaversion-than-my-system

build.sbt 不适用于不同的 Scala 版本 https://stackoverflow.com/questions/75883832/build-sbt-does-not-work-with-different-scala-versions