无法在运行时修改mapred.job.name。它不在允许在运行时修改的参数列表中

2023-11-29

我正在尝试在气流中运行一些蜂巢作业。我建立了custome jdbc 连接，您可以在图中看到。我可以通过 Airflow Web ui 查询 Hive 表（数据分析 -> 即席查询）。我还想运行一些来自互联网的示例 dag 文件：

#File Name: wf_incremental_load.py
from airflow import DAG
from airflow.operators import BashOperator, HiveOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2019, 3, 13),
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG('hive_test', default_args=default_args,schedule_interval='* */5 * * *')

touch_job = """
 touch /root/hive.txt
"""
# Importing the data from Mysql table to HDFS
task1 = BashOperator(
        task_id= 'make_file',
        bash_command=touch_job,
        dag=dag
)

# Inserting the data from Hive external table to the target table
task2 = HiveOperator(
        task_id= 'hive_table_create',
        hql='CREATE TABLE aaaaa AS SELECT * FROM ant_code;',
    hive_cli_conn_id='hive_jdbc',
        depends_on_past=True,
        dag=dag
)
# defining the job dependency
task2.set_upstream(task1)

然而，当我在气流中运行这项工作时，我遇到了一些错误：

错误 jdbc.Utils：无法从 ZooKeeper 读取 HiveServer2 配置错误：无法在 ZooKeeper 中打开任何服务器 URI 的客户端传输：无法打开新会话：java.lang.IllegalArgumentException：无法在运行时修改 mapred.job.name。它不在允许在运行时修改的参数列表中（state=08S01，code=0）直线 > 使用默认值；当前没有连接

[2019-03-13 13:32:25,335] {models.py:1593} INFO - Executing <Task(HiveOperator): hive_table_create> on 2019-03-13T00:00:00+00:00
[2019-03-13 13:32:25,336] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run hive_test hive_table_create 2019-03-13T00:00:00+00:00 --job_id 19 --raw -sd DAGS_FOLDER/hive_test.py --cfg_path /tmp/tmphSGJhO']
[2019-03-13 13:32:27,130] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create [2019-03-13 13:32:27,129] {__init__.py:51} INFO - Using executor SequentialExecutor
[2019-03-13 13:32:27,548] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create [2019-03-13 13:32:27,547] {models.py:273} INFO - Filling up the DagBag from /root/airflow/dags/hive_test.py
[2019-03-13 13:32:27,565] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create /usr/lib/python2.7/site-packages/airflow/utils/helpers.py:356: DeprecationWarning: Importing 'BashOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2019-03-13 13:32:27,565] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   DeprecationWarning)
[2019-03-13 13:32:27,570] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create /usr/lib/python2.7/site-packages/airflow/utils/helpers.py:356: DeprecationWarning: Importing 'HiveOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2019-03-13 13:32:27,570] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   DeprecationWarning)
[2019-03-13 13:32:27,602] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create [2019-03-13 13:32:27,602] {cli.py:520} INFO - Running <TaskInstance: hive_test.hive_table_create 2019-03-13T00:00:00+00:00 [running]> on host name02.excard.co.kr
[2019-03-13 13:32:27,625] {hive_operator.py:118} INFO - Executing: CREATE TABLE aaaaa AS SELECT * FROM ant_code;
[2019-03-13 13:32:27,634] {logging_mixin.py:95} INFO - [2019-03-13 13:32:27,634] {base_hook.py:83} INFO - Using connection to: id: hive_jdbc. Host: jdbc:hive2://192.168.0.202:10000/big_info, Port: None, Schema: None, Login: hive, Password: XXXXXXXX, extra: {u'extra__jdbc__drv_path': u'/usr/hdp/3.1.0.0-78/hive/jdbc/hive-jdbc-3.1.0.3.1.0.0-78-standalone.jar', u'extra__google_cloud_platform__scope': u'', u'extra__google_cloud_platform__project': u'', u'extra__google_cloud_platform__key_path': u'', u'extra__jdbc__drv_clsname': u'org.apache.hive.jdbc.HiveDriver', u'extra__google_cloud_platform__keyfile_dict': u''}
[2019-03-13 13:32:27,636] {hive_operator.py:133} INFO - Passing HiveConf: {'airflow.ctx.task_id': 'hive_table_create', 'airflow.ctx.dag_id': 'hive_test', 'airflow.ctx.execution_date': '2019-03-13T00:00:00+00:00', 'airflow.ctx.dag_run_id': u'scheduled__2019-03-13T00:00:00+00:00'}
[2019-03-13 13:32:27,637] {logging_mixin.py:95} INFO - [2019-03-13 13:32:27,637] {hive_hooks.py:236} INFO - hive -hiveconf airflow.ctx.task_id=hive_table_create -hiveconf airflow.ctx.dag_id=hive_test -hiveconf airflow.ctx.execution_date=2019-03-13T00:00:00+00:00 -hiveconf airflow.ctx.dag_run_id=scheduled__2019-03-13T00:00:00+00:00 -hiveconf mapred.job.name=Airflow HiveOperator task for name02.hive_test.hive_table_create.2019-03-13T00:00:00+00:00 -f /tmp/airflow_hiveop_rXXLyV/tmpdZYjMS
[2019-03-13 13:32:32,323] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,323] {hive_hooks.py:251} INFO - Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
[2019-03-13 13:32:32,738] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,738] {hive_hooks.py:251} INFO - 19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
[2019-03-13 13:32:32,813] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,813] {hive_hooks.py:251} INFO - 19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
[2019-03-13 13:32:32,830] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,830] {hive_hooks.py:251} INFO - 19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://name02:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
[2019-03-13 13:32:32,895] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,895] {hive_hooks.py:251} INFO - 19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
[2019-03-13 13:32:32,941] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,941] {hive_hooks.py:251} INFO - 19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
[2019-03-13 13:32:32,959] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,959] {hive_hooks.py:251} INFO - 19/03/13 13:32:32 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
[2019-03-13 13:32:32,967] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,967] {hive_hooks.py:251} INFO - Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
[2019-03-13 13:32:32,980] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,980] {hive_hooks.py:251} INFO - beeline> USE default;
[2019-03-13 13:32:32,988] {logging_mixin.py:95} INFO - [2019-03-13 13:32:32,988] {hive_hooks.py:251} INFO - No current connection
[2019-03-13 13:32:33,035] {models.py:1788} ERROR - Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://name02:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
19/03/13 13:32:32 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
beeline> USE default;
No current connection
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/lib/python2.7/site-packages/airflow/operators/hive_operator.py", line 134, in execute
    self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
  File "/usr/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py", line 255, in run_cli
    raise AirflowException(stdout)
AirflowException: Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://name02:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
19/03/13 13:32:32 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
beeline> USE default;
No current connection

[2019-03-13 13:32:33,037] {models.py:1817} INFO - All retries failed; marking task as FAILED
[2019-03-13 13:32:33,546] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create Traceback (most recent call last):
[2019-03-13 13:32:33,546] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/bin/airflow", line 32, in <module>
[2019-03-13 13:32:33,547] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     args.func(args)
[2019-03-13 13:32:33,547] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-03-13 13:32:33,547] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     return f(*args, **kwargs)
[2019-03-13 13:32:33,547] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 526, in run
[2019-03-13 13:32:33,547] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     _run(args, dag, ti)
[2019-03-13 13:32:33,548] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 445, in _run
[2019-03-13 13:32:33,548] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     pool=args.pool,
[2019-03-13 13:32:33,548] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 73, in wrapper
[2019-03-13 13:32:33,548] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     return func(*args, **kwargs)
[2019-03-13 13:32:33,548] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
[2019-03-13 13:32:33,549] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     result = task_copy.execute(context=context)
[2019-03-13 13:32:33,549] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/operators/hive_operator.py", line 134, in execute
[2019-03-13 13:32:33,549] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
[2019-03-13 13:32:33,549] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create   File "/usr/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py", line 255, in run_cli
[2019-03-13 13:32:33,549] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create     raise AirflowException(stdout)
[2019-03-13 13:32:33,550] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create airflow.exceptions.AirflowException: Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
[2019-03-13 13:32:33,550] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
[2019-03-13 13:32:33,550] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
[2019-03-13 13:32:33,551] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://name02:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
[2019-03-13 13:32:33,551] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 19/03/13 13:32:32 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
[2019-03-13 13:32:33,551] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 19/03/13 13:32:32 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
[2019-03-13 13:32:33,552] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 19/03/13 13:32:32 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
[2019-03-13 13:32:33,552] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
[2019-03-13 13:32:33,552] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create beeline> USE default;
[2019-03-13 13:32:33,552] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create No current connection
[2019-03-13 13:32:33,552] {base_task_runner.py:101} INFO - Job 19: Subtask hive_table_create 
[2019-03-13 13:32:35,201] {logging_mixin.py:95} INFO - [2019-03-13 13:32:35,201] {jobs.py:2527} INFO - Task exited with return code 1

请帮忙解决问题。

Update:

我在 hive-site.xml 中添加 hive.security.authorization.sqlstd.confwhitelist.append : mapred.job.name* 。

所以现在我遇到了一些不同的错误：

错误：无法在 ZooKeeper 中打开任何服务器 URI 的客户端传输：无法打开新会话：java.lang.IllegalArgumentException：无法在运行时修改airflow.ctx.task_id。它不在允许在运行时修改的参数列表中（state=08S01，code=0）直线 > 使用默认值；当前没有连接回溯（最近一次调用最后一次）：

[2019-03-13 14:54:31,946] {models.py:1593} INFO - Executing <Task(HiveOperator): hive_table_create> on 2019-03-13T00:00:00+00:00
[2019-03-13 14:54:31,947] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run hive_test hive_table_create 2019-03-13T00:00:00+00:00 --job_id 11 --raw -sd DAGS_FOLDER/hive_test.py --cfg_path /tmp/tmpGDjT7j']
[2019-03-13 14:54:33,793] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create [2019-03-13 14:54:33,792] {__init__.py:51} INFO - Using executor SequentialExecutor
[2019-03-13 14:54:34,189] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create [2019-03-13 14:54:34,189] {models.py:273} INFO - Filling up the DagBag from /root/airflow/dags/hive_test.py
[2019-03-13 14:54:34,192] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create /usr/lib/python2.7/site-packages/airflow/utils/helpers.py:356: DeprecationWarning: Importing 'BashOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2019-03-13 14:54:34,193] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create   DeprecationWarning)
[2019-03-13 14:54:34,195] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create /usr/lib/python2.7/site-packages/airflow/utils/helpers.py:356: DeprecationWarning: Importing 'HiveOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2019-03-13 14:54:34,195] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create   DeprecationWarning)
[2019-03-13 14:54:34,219] {base_task_runner.py:101} INFO - Job 11: Subtask hive_table_create [2019-03-13 14:54:34,218] {cli.py:520} INFO - Running <TaskInstance: hive_test.hive_table_create 2019-03-13T00:00:00+00:00 [running]> on host name02.excard.co.kr
[2019-03-13 14:54:34,240] {hive_operator.py:118} INFO - Executing: CREATE TABLE aaaaa AS SELECT * FROM ant_code;
[2019-03-13 14:54:34,249] {logging_mixin.py:95} INFO - [2019-03-13 14:54:34,249] {base_hook.py:83} INFO - Using connection to: id: hive_jdbc. Host: jdbc:hive2://192.168.0.202:10000/big_info, Port: None, Schema: None, Login: hive, Password: XXXXXXXX, extra: {u'extra__jdbc__drv_path': u'/usr/hdp/3.1.0.0-78/hive/jdbc/hive-jdbc-3.1.0.3.1.0.0-78-standalone.jar', u'extra__google_cloud_platform__scope': u'', u'extra__google_cloud_platform__project': u'', u'extra__google_cloud_platform__key_path': u'', u'extra__jdbc__drv_clsname': u'org.apache.hive.jdbc.HiveDriver', u'extra__google_cloud_platform__keyfile_dict': u''}
[2019-03-13 14:54:34,251] {hive_operator.py:133} INFO - Passing HiveConf: {'airflow.ctx.task_id': 'hive_table_create', 'airflow.ctx.dag_id': 'hive_test', 'airflow.ctx.execution_date': '2019-03-13T00:00:00+00:00', 'airflow.ctx.dag_run_id': u'scheduled__2019-03-13T00:00:00+00:00'}
[2019-03-13 14:54:34,253] {logging_mixin.py:95} INFO - [2019-03-13 14:54:34,252] {hive_hooks.py:236} INFO - hive -hiveconf airflow.ctx.task_id=hive_table_create -hiveconf airflow.ctx.dag_id=hive_test -hiveconf airflow.ctx.execution_date=2019-03-13T00:00:00+00:00 -hiveconf airflow.ctx.dag_run_id=scheduled__2019-03-13T00:00:00+00:00 -hiveconf mapred.job.name=Airflow HiveOperator task for name02.hive_test.hive_table_create.2019-03-13T00:00:00+00:00 -f /tmp/airflow_hiveop_wNbQlL/tmpFN6MGy
[2019-03-13 14:54:39,061] {logging_mixin.py:95} INFO - [2019-03-13 14:54:39,060] {hive_hooks.py:251} INFO - Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
[2019-03-13 14:54:39,443] {logging_mixin.py:95} INFO - [2019-03-13 14:54:39,443] {hive_hooks.py:251} INFO - 19/03/13 14:54:39 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
[2019-03-13 14:54:39,532] {logging_mixin.py:95} INFO - [2019-03-13 14:54:39,532] {hive_hooks.py:251} INFO - 19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
[2019-03-13 14:54:39,552] {logging_mixin.py:95} INFO - [2019-03-13 14:54:39,551] {hive_hooks.py:251} INFO - 19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://data01:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify airflow.ctx.task_id at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
[2019-03-13 14:54:39,664] {logging_mixin.py:95} INFO - [2019-03-13 14:54:39,664] {hive_hooks.py:251} INFO - 19/03/13 14:54:39 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
[2019-03-13 14:54:39,856] {logging_mixin.py:95} INFO - [2019-03-13 14:54:39,856] {hive_hooks.py:251} INFO - 19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
[2019-03-13 14:54:41,134] {logging_mixin.py:95} INFO - [2019-03-13 14:54:41,134] {hive_hooks.py:251} INFO - 19/03/13 14:54:41 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
[2019-03-13 14:54:41,147] {logging_mixin.py:95} INFO - [2019-03-13 14:54:41,146] {hive_hooks.py:251} INFO - Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify airflow.ctx.task_id at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
[2019-03-13 14:54:41,167] {logging_mixin.py:95} INFO - [2019-03-13 14:54:41,167] {hive_hooks.py:251} INFO - beeline> USE default;
[2019-03-13 14:54:41,180] {logging_mixin.py:95} INFO - [2019-03-13 14:54:41,180] {hive_hooks.py:251} INFO - No current connection
[2019-03-13 14:54:41,253] {models.py:1788} ERROR - Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
19/03/13 14:54:39 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://data01:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify airflow.ctx.task_id at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
19/03/13 14:54:39 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
19/03/13 14:54:41 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify airflow.ctx.task_id at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
beeline> USE default;
No current connection
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/lib/python2.7/site-packages/airflow/operators/hive_operator.py", line 134, in execute
    self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
  File "/usr/lib/python2.7/site-packages/airflow/hooks/hive_hooks.py", line 255, in run_cli
    raise AirflowException(stdout)
AirflowException: Connecting to jdbc:hive2://name01.excard.co.kr:2181,name02.excard.co.kr:2181,data01.excard.co.kr:2181/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
19/03/13 14:54:39 [main]: INFO jdbc.HiveConnection: Connected to data01:10000
19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Failed to connect to data01:10000
19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://data01:10000/default;password=root;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify airflow.ctx.task_id at runtime. It is not in list of params that are allowed to be modified at runtime Retrying 0 of 1
19/03/13 14:54:39 [main]: INFO jdbc.HiveConnection: Connected to name02:10000
19/03/13 14:54:39 [main]: WARN jdbc.HiveConnection: Failed to connect to name02:10000
19/03/13 14:54:41 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify airflow.ctx.task_id at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
beeline> USE default;
No current connection

请添加到hive.security.authorization.sqlstd.confwhitelist.append下面列出了参数：

airflow.ctx.dag_id
airflow.ctx.task_id
airflow.ctx.execution_date
airflow.ctx.dag_run_id
airflow.ctx.dag_owner
airflow.ctx.dag_email
mapred.job.name

or just

airflow.ctx.*
mapred.job.name

默认情况下，Airflow 在运行时修改这些参数。这应该有效。

本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)

无法在运行时修改mapred.job.name。它不在允许在运行时修改的参数列表中的相关文章

如何编写嵌套的 __init__.py 文件

我正在努力解决嵌套问题 init py在我正在编写的Python包中该包具有以下架构 module init py submodule1 init py source py submodule2 init py source py sub
如何在多进程系统中实现锁定？

我们正在并行运行许多詹金斯项目我们使用 python 并且选择使用 pyenv 管理虚拟环境不幸的是 pyenv 有一个众所周知的竞争条件 https github com yyuu pyenv issues 174 为了解决这个问题
此 TypeError 消息中提到的“代码对象”是什么？

在尝试使用Python时exec声明我收到以下错误 TypeError exec arg 1 must be a string file or code object 我不想传递字符串或文件但什么是代码对象如何创建一个创建代码对象的
Virtualenv 在 OS X Yosemite 上失败并出现 OSError

我最近更新到 OSX Yosemite 现在无法使用virtualenv pip 每当我执行 virtualenv env 它抛出一个 OSError Command Users administrator ux env bin pytho
使用 django-rest-framework 设置对象级权限

尝试使用 django rest framework 最干净最规范地管理 django guardian 对象级权限我想将对象的读取权限 module view object 分配给在执行 POST 时发出请求的用户我的基于阶级的观点
将非常大的Python列表输出保存到mysql表中

我想将 python 生成的列表的输出保存在 mysql 数据库的表中该表如下所示 mysql 中的 myapc8 表 https i stack imgur com 4B4Hz png这是Python代码在此输入图像描述 https
在python中将数据库表写入文件的最快方法

我正在尝试从数据库中提取大量数据并将其写入 csv 文件我正在尝试找出最快的方法来做到这一点我发现在 fetchall 的结果上运行 writerows 比下面的代码慢 40 with open filename a as f writ
无法在 selenium 和 requests 之间传递 cookie，以便使用后者进行抓取

我用 python 结合 selenium 编写了一个脚本来登录网站然后从driver to requests这样我就可以继续使用requests进行进一步的活动 I used item soup select one div class
python 中的 <> 运算符有什么作用？

我刚刚遇到这个here http www feedparser org feedparser py 总是这样使用 if string1 find string2 lt gt 1 pass 什么是 lt gt 运算符这样做为什么不使用通常的
查找 Pandas DF 行中的最短日期并创建新列

我有一个包含多个日期的表有些日期将为 NaN 我需要找到最旧的日期所以一行可能有 DATE MODIFIED WITHDRAWN DATE SOLD DATE STATUS DATE 等因此对于每一行一个或多个字段中都会有一个日期
Python bug - 或者我的愚蠢 - 扫描字符串文字时 EOL

我看不出以下两行之间有显着差异然而第一个解析而后者则不解析 In 5 n Axis of Awesome In 6 n Axis of Awesome File
如何在亚马逊 EC2 上调试 python 网站？

我是网络开发新手这可能是一个愚蠢的问题但我找不到可以帮助我的确切答案或教程我工作的公司的网站用 python django 构建托管在亚马逊 EC2 上我想知道从哪里开始调试这个生产站点并检查存储在那里的日志和数据库我有帐户信
minizinc python 安装

我通过 anaconda 提示符在 python 上安装了 minizinc 就像其他软件包一样 pip install minizinc 该软件包表示已成功安装我可以导入该模块但是我正在遵循基本示例https minizinc py
Django 管理器链接

我想知道是否有可能如果可以的话如何将多个管理器链接在一起以生成受两个单独管理器影响的查询集我将解释我正在研究的具体示例我有多个抽象模型类用于为其他模型提供小型的特定功能其中两个模型是DeleteMixin 和GlobalMix
带 Flask 的 RPI dht22：无法将第 4 行设置为输入 - 等待 PulseIn 消息超时

我正在尝试制作一个 Raspberry Pi 3 REST API 使用 DHT22 提供温度和湿度整个代码 from flask import Flask jsonify request from sds011 import SDS01
如何从namedtuple实例列表创建pandas DataFrame（带有索引或多索引）？

简单的例子 from collections import namedtuple import pandas Price namedtuple Price ticker date price a Price GE 2010 01 01 30
将 Keras 集成到 SKLearn 管道？

我有一个 sklearn 管道对异构数据类型布尔分类数字文本执行特征工程并想尝试使用神经网络作为我的学习算法来拟合模型我遇到了输入数据形状的一些问题我想知道我想做的事情是否可能或者我是否应该尝试不同的方法我尝试了几种不
IndexError - 具有匀称形状的笛卡尔 PolygonPatch

我曾经使用 shapely 制作一个圆圈并将其绘制在之前填充的图上这曾经工作得很好最近我收到索引错误我将代码分解为最简单的操作但它甚至无法执行最简单的循环 import descartes import shapely geome
如何（安全）将 Python 对象发送到我的 Flask API？

我目前正在尝试构建一个 Flask Web API 它能够在 POST 请求中接收 python 对象我使用 Python 3 7 1 创建请求使用 Python 2 7 运行 API 该 API 设置为在我的本地计算机上运行我试图发
无法安装最新版本的 Numpy (1.22.3)

我正在尝试安装最新版本的 numpy 即 1 22 3 但看起来 pip 无法找到最后一个版本我知道我可以从源代码本地安装它但我想了解为什么我无法使用 pip 安装它 PS 我有最新版本的pip 22 0 4 ERROR Could n

随机推荐

Excel 文件被覆盖而不是连接 - Python - Pandas

我正在尝试使用以下脚本将其中的所有 Excel 文件和工作表合并为一个它有点工作但随后 Excel 文件 c xlsx 被每个文件覆盖因此只有最后一个 Excel 文件被连接不知道为什么 import pandas as pd im
grails 模板 - 脚手架控制器

我是 grails 的新手我现在正在研究我的脚手架模板特别是我的控制器每次生成它时我都想要一个定制的控制器所以我使用了安装模板我总是在控制器上创建一个命令对象是否可以将域类中的字段包含到生成的控制器中的命令对象中我知道我必须
Java - 如何从哈希图中找到最接近特定数字的值？

嗨我有一个HashMap
Javascript：使用 setTimeout 重试的函数

我有一个函数downloadItem由于网络原因可能会失败我希望能够在实际拒绝该项目之前重试几次重试需要超时因为如果出现网络问题则立即重试是没有意义的这是我到目前为止所拥有的 function downloadItemWithRe
这段 javascript 行中的冒号 (:) 是什么意思？

下面第3 6行中的是什么意思 function displayError error var errorTypes 0 Unknown error 1 Permission denied 2 Position is not availab
使用 Asynctask 在列表视图中显示数据

我成功在列表视图中显示了来自 Web 服务 JSON 的数据但我想添加 Asyntask 我可以在我的代码中放置代码 Asyntask 的位置这是我的代码用于在列表视图中显示数据 public class Jadwal remix e
JQuery 数据表中的 TableTools 导出不起作用

我已经为此苦苦挣扎了两天我已经在数据表论坛上发帖了但没有得到回复所以我希望这里的社区能够提供帮助我在这里阅读过有关此问题的类似帖子但正如许多帖子所建议的那样我已经使用了 sSwf 的正确路径出现导出按钮并加载所有资源但是单
UITableViewCell 无法用一根手指点击来点击，但可以用两根手指点击

我创建了一个表视图并且 tableViewCell 无法用一根手指单击但是当我尝试用两根手指单击 tableViewCell 时会发生单击事件我不知道为什么会出现这种情况我在 tableView 中创建了一个自定义单元格 Invi
将公式转换为 CNF

有没有办法使用 z3 将公式转换为 CNF 使用 Tseitsin 式编码我正在寻找类似的东西simplify命令但保证返回的公式为 CNF 您可以使用apply命令来执行此操作我们可以为该命令提供任意战术策略有关 Z3 4 0
Crystal Reports 打印纵向而不是横向

我在 SAP Crystal Reports 14 1 4 1327 中创建了一个非常简单的测试报告 rpt 文件我在页面设置中将页面方向设置为横向如果我从 Crystal Reports Designer 打印文档它将以横向方式正确
如何在Form1、WinForms C#中使用Form2的变量？

我在 Visual Studio 2013 中有一个解决方案其中包含两个表单我想要当在 Form2 中按下按钮时变量flag fb已更新我在 Form1 中使用它的值有什么办法可以做到这一点吗谢谢方法一使用参数化构造函数在表
Android：PatternSyntaxException：语法错误 U_ILLEGAL_ARGUMENT_ERROR

我在用着json io将一些 JSON 数据转换为 Java 对象 DataTransferContainer dataTransferContainer DataTransferContainer JsonReader jsonToJav
使用 gnuplot 绘制两个相邻的“带框”图

我试图将两个数据系列绘制在一张图中作为 gnuplot 中的方框该脚本目前如下所示 set terminal postscript eps enhanced color set title Distribution of the ext
C++：调用派生类的虚函数

假设我有一个带有虚函数的类和一个以不同方式实现该虚函数的派生类假设我还有一个用于存储派生类的基类向量如果事先不知道派生类是什么我将如何在向量中执行派生类的虚函数说明问题的最少代码 include
如何在任意 Java 应用程序中启用抗锯齿功能？

尽管 Swing 能够提供抗锯齿字体但许多 Java 应用程序默认情况下并不使用抗锯齿字体如何强制任意 Java 应用程序使用 AA 字体适用于我正在运行的应用程序和我正在开发的应用程序如果您有权访问源代码则可以在 main 方法
创建色轮的功能[关闭]

Closed 这个问题需要多问focused 目前不接受答案这是我多次伪解决但从未完全找到解决方案的问题问题是想出一种方法来生成N颜色尽可能区分N是一个参数我对此的第一个想法是如何在空间中生成 N 个向量使彼此之间的距离最大化
单元格悬停时的工具提示图像，Google 电子表格

我目前正在制作一个必须包含大量图像引用的 Google 电子表格我想要实现的是当您将鼠标悬停在单元格上时会出现图像如工具提示 I found 这个小部件在 Google Developers 上但是当我将代码添加到电子表格时什么也
IE9 下的 FakeSmile

我正在尝试在 IE9 中伪造 SMIL 支持假笑我动态创建 SVG 元素添加带有 animate 元素的 rect 元素并调用 beginElement IE9 给我一个错误对象不支持属性或方法 beginElement 静态 SVG
在数据库中存储标签的最有效方法是什么？

我正在我的网站上实现一个类似于 stackoverflow 使用的标签系统我的问题是存储标签以便搜索和过滤标签的最有效方法是什么我的想法是这样的 Table Items Columns Item ID Title Content Ta
无法在运行时修改mapred.job.name。它不在允许在运行时修改的参数列表中

我正在尝试在气流中运行一些蜂巢作业我建立了custome jdbc 连接您可以在图中看到我可以通过 Airflow Web ui 查询 Hive 表数据分析 gt 即席查询我还想运行一些来自互联网的示例 dag 文件 File N

无法在运行时修改mapred.job.name。它不在允许在运行时修改的参数列表中

无法在运行时修改mapred.job.name。它不在允许在运行时修改的参数列表中 的相关文章

随机推荐

热门标签

无法在运行时修改mapred.job.name。它不在允许在运行时修改的参数列表中的相关文章