我目前使用的是Python：3.7.9，spark：spark-2.4.6-bin-hadoop2.6
在这个项目 venv 中，我的设置为：
kafka-python 2.0.2
pip 21.2.4
py4j 0.10.9
pyspark 2.4.6
setuptools 57.4.0
系统环境变量：
SPARK_HOME C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6
PYTHONPATH %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src.zip;%PYTHONPATH%
PATH %SPARK_HOME%\bin;%SPARK_HOME%\python;%PATH%
JAVA_HOME C:\Program Files\Java\jdk1.8.0_261
我正在尝试在 Python 中 Spark 提交以下代码：
import sys
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
from kafka import KafkaConsumer
kafka_consumer.py 位于 C:\Users\MANU\PycharmProjects\pythonProject 1
Command I ran: (venv) C:\Users\MANU\PycharmProjects\pythonProject1>spark-submit kafka_consumer.py
我在控制台中得到这个：
Traceback (most recent call last):
File "C:/Users/MANU/PycharmProjects/pythonProject1/kafka_consumer.py", line 1, in <module>
from pyspark import SparkContext, SparkConf
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\__init__.py", line 51, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 31, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\accumulators.py", line 97, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\serializers.py", line 72, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\cloudpickle.py", line 145, in <module>
File "C:\spark-2.4.6-bin-hadoop2.6\spark-2.4.6-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
TypeError: an integer is required (got type bytes)
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
