我认识以及改造flink的第一步从下载编译源码开始
0、环境
idea
java 1.8
scala 2.12
flink 1.16
maven-setting(核心)
nodejs(这个去官网下载安装一下就好,npm -v 显示成功就好)
文末附有核心setting的配置
1、下载源码
由于网络问题,所以我是从gitee上进行代码的拉取
flink: Apache Flink 是高效和分布式的通用数据处理平台
2、idea 导入源码
记住,导入源码后要检查四个问题:
idea plugin中scala是否设置了(可能会有网络问题,设置一下代理,代理如下,设置前ping一下,失效了再去搜一个对应网址地址,设置完毕后重启一下idea)
13.249.171.117 plugins.jetbrains.com
13.249.171.117 jetbrains.com
scala sdk环境是否设置了;
java sdk是否设置了;
maven 环境是否设置了
3、编译打包
根目录下运行,运行完毕后可能要等30min左右(起码我是),然后最后会显示build sucess;假如有特殊需求需要跳过某一个,则可以执行 mvn -rf XXXModule
测试代码不能跳过编译,但可跳过执行
mvn install -Drat.skip=true -DskipTests -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfast -T 4 -Dmaven.compile.fork=true
4、测试运行
运行exapmple中的stream-wordcount,测试编译等是否正常;正常会显示最终的wordcount结果,如下所示
在测试过程中可能会有问题,经常出现的一个问题是akka抛错,可以根据后面的提示进行解决,如下:
mvn clean package -pl flink-rpc/flink-rpc-akka,flink-rpc/flink-rpc-akka-loader -DskipTests
附件
setting文件-setting.xml
注意本地仓库的地址要记得修改为自己的地址,其他照搬即可
<?xml version="1.0" encoding="utf-8"?>
<settings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<!--本地仓库路径-->
<localRepository>E:\maven-repository</localRepository>
<!--注意:mirrorOf千万别为*,建议为profile的id-->
<mirrors>
<mirror>
<id>aliyun-nexus</id>
<name>aliyun-nexus</name>
<mirrorOf>aliyun</mirrorOf>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
<mirror>
<id>huawei-nexus</id>
<name>huawei-nexus</name>
<mirrorOf>huawei</mirrorOf>
<url>https://mirrors.huaweicloud.com/repository/maven/</url>
</mirror>
<mirror>
<id>confluent-nexus</id>
<name>confluent-nexus</name>
<mirrorOf>confluent</mirrorOf>
<url>https://packages.confluent.io/maven/</url>
</mirror>
</mirrors>
<profiles>
<profile>
<id>aliyun</id>
<repositories>
<repository>
<id>aliyun-nexus</id>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>aliyun-nexus</id>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
<profile>
<id>huawei</id>
<repositories>
<repository>
<id>huawei-nexus</id>
<url>https://mirrors.huaweicloud.com/repository/maven/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>huawei-nexus</id>
<url>https://mirrors.huaweicloud.com/repository/maven/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
<profile>
<id>confluent</id>
<repositories>
<repository>
<id>confluent-nexus</id>
<url>https://packages.confluent.io/maven/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>confluent-nexus</id>
<url>https://packages.confluent.io/maven/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
<profile>
<id>cdh</id>
<repositories>
<repository>
<id>cdh-nexus</id>
<releases>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
<checksumPolicy>fail</checksumPolicy>
</releases>
<snapshots>
<enabled>false</enabled>
<updatePolicy>always</updatePolicy>
<checksumPolicy>warn</checksumPolicy>
</snapshots>
<name>Cloudera Repositories</name>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<layout>default</layout>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>cdh-nexus</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
<profile>
<id>apache</id>
<repositories>
<repository>
<id>apache-nexus</id>
<releases>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
<checksumPolicy>fail</checksumPolicy>
</releases>
<snapshots>
<enabled>false</enabled>
<updatePolicy>always</updatePolicy>
<checksumPolicy>warn</checksumPolicy>
</snapshots>
<name>apache Repositories</name>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<layout>default</layout>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>apache-nexus</id>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
<!--设置默认配置文件-->
<activeProfiles>
<activeProfile>aliyun</activeProfile>
</activeProfiles>
</settings>
参考:
1、flink编译过程中遇到的问题以及加速 https://blog.csdn.net/spark_dev/article/details/124008534