着色规则仅在您构建胖罐子时适用。它不会在其他 sbt 任务中应用。
如果您想对 hadoop 依赖项内部的某个库进行遮蔽，您可以创建一个仅包含 hadoop 依赖项的新项目，对库进行遮蔽，然后发布一个包含所有遮蔽的 hadoop 依赖项的 fat jar。
这不是一个完美的解决方案，因为新的 hadoop jar 中的所有依赖项对于谁使用它们来说都是“未知的”，并且您需要手动处理冲突。
这是您在您的应用程序中需要的代码build.sbt发布一个胖hadoop jar
（使用你的代码和 sbt 程序集docs https://github.com/sbt/sbt-assembly#publishing-not-recommended):
val HadoopVersion = "2.6.0-cdh5.11.0"
val hadoopHdfs = "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion
val hadoopCommon = "org.apache.hadoop" % "hadoop-common" % HadoopVersion
val hadoopHdfsTest = "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion classifier "tests"
val hadoopCommonTest = "org.apache.hadoop" % "hadoop-common" % HadoopVersion % classifier "tests"
val hadoopMiniDFSCluster = "org.apache.hadoop" % "hadoop-minicluster" % HadoopVersion
lazy val fatJar = project
.enablePlugins(AssemblyPlugin)
.settings(
libraryDependencies ++= Seq(
hadoopHdfs,
hadoopCommon,
hadoopHdfsTest,
hadoopCommonTest,
hadoopMiniDFSCluster
),
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.common.**" -> "shade.@0").inAll
),
assemblyMergeStrategy in assembly := {
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.first
},
artifact in (Compile, assembly) := {
val art = (artifact in (Compile, assembly)).value
art.withClassifier(Some("assembly"))
},
addArtifact(artifact in (Compile, assembly), assembly),
crossPaths := false, // Do not append Scala versions to the generated artifacts
autoScalaLibrary := false, // This forbids including Scala related libraries into the dependency
skip in publish := true
)
lazy val shaded_hadoop = project
.settings(
name := "shaded-hadoop",
packageBin in Compile := (assembly in (fatJar, Compile)).value
)
我还没有测试过，但这就是要点。
我想指出我注意到的另一个问题，您的合并策略可能会给您带来问题，因为您想对某些文件应用不同的策略。查看默认策略here https://github.com/sbt/sbt-assembly#merge-strategy.
我建议使用类似的方法来保留所有不适用的内容的原始策略deduplicate
assemblyMergeStrategy in assembly := {
entry: String => {
val strategy = (assemblyMergeStrategy in assembly).value(entry)
if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
else strategy
}
}
