Hadoop 2.6.0
Flink 1.6.0
Hive 1.1.0
Python 2.7.0
下载Hadoop2.6.0, HadoopWinutils , Spark2.0.2资源包
下载网址:
hadoop
https://archive.apache.org/dist/hadoop/common/
hadoopmaster
https://github.com/4ttty/winutils
spark
https://archive.apache.org/dist/spark/
第一步 将这些资源包下载好后,都解压放入到英文目录下
将hadooponwindows-master/bin目录下的hadoop.dll和winutils.exe文件放入到hadoop2.6.0/bin
第二步 下载并安装Anaconda3,直接上官网下载就可以
使用Anaconda 安装Python2.7.0
第三步Hive配置文件修改
hive-site.xml中配置如下,取消spark和hive之间的版本校验,因为spark-home/jars目录下的hive版本是1.2.0,而CDH5.11安装的版本是1.1.0版本,存在不同
然后重启hive
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in metastore matches with one from Hive jars. Also disable automatic
schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures
proper metastore schema migration. (Default)
False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
修改hdfs-site.xml,关闭权限校验
<property>
<name>dfs.permissions</name>
<value>false</value>
<description>
If "true", enable permission checking in HDFS.
If "false", permission checking is turned off,
but all other behavior is unchanged.
Switching from one parameter value to the other does not change the mode,
owner or group of files or directories.
</description>
</property>
或者直接找到hadoop的权限用户–hdfs,在环境中进行配置HADOOP_USER_NAME
linux集群某个节点,登录上,找到DNS解析配置文件
vim /etc/hosts
将其中的配置IP,复制粘贴到windows系统的hosts文件中
C:\Windows\System32\drivers\etc\hosts
将Linux集群中的配置文件hive-site.xml , hdfs-site.xml , core-site.xml
粘贴到windows环境下的Spark_Home/conf, Hadoop_Home/etc/hadoop/ 的目录下
conda create -n py2.7 python==2.7.0
conda activate py2.7
conda intsall pyspark==2.2.1
win+r 输入cmd ,然后 回车
接着测试配置的系统环境
hadoop
python --version
pyspark
使用idea进行创建,需要使用python插件,可去idea官网下载使用
以下是python代码
from pyspark.sql import HiveContext, SparkSession
_APP_NAME = "test"
spark_session = SparkSession.builder\
.master("local")\
.config("hive.metastore.uris", "thrift://172.10.1.11:9083")\
.appName(_APP_NAME).enableHiveSupport().getOrCreate()
# 通过SQL语句在hive中查询的数据直接是dataframe的形式
read_df = spark_session.sql("show databases").show()
idea的环境并没有读取到一些环境变量,需要手动添加
PYTHONPATH=SPARK_HOME/python
