没有这样的文件或目录 /airflow/xcom/return.json

2024-04-12

创建了一个图像包含/airflow/xcom/return.json在所有子目录上使用 chmod +x 由于日志显示找不到文件或目录(尝试过 chmod +x)

strtpodbefore = KubernetesPodOperator(namespace='rguonew',
                                    image="mydockerimage",
                                    name="fail",
                                    image_pull_policy="Always",
                                    task_id="failing-task",
                                    get_logs= True,
                                    xcom_push=True,
                                    dag=dag
                                    )'

这是日志

[2019-03-18 20:32:07,007] {logging_mixin.py:95} INFO - [2019-03-18 
 20:32:07,007] {pod_launcher.py:166} INFO - Running command... cat 
/airflow/xcom/return.json
[2019-03-18 20:32:07,007] {logging_mixin.py:95} INFO - 
[2019-03-18 20:32:07,026] {logging_mixin.py:95} INFO - [2019-03-18 
20:32:07,026] {pod_launcher.py:173} INFO - cat: can't open 
 '/airflow/xcom/return.json': No such file or directory
[2019-03-18 20:32:07,026] {logging_mixin.py:95} INFO - 
[2019-03-18 20:32:07,026] {logging_mixin.py:95} INFO - [2019-03-18 
 20:32:07,026] {pod_launcher.py:166} INFO - Running command... kill -s 
 SIGINT 1
[2019-03-18 20:32:07,026] {logging_mixin.py:95} INFO - 
[2019-03-18 20:32:07,067] {models.py:1788} ERROR - Pod Launching 
failed: Failed to extract xcom from pod: fail-e18e3dac

所以尝试用这种方式工作,但这意味着它从外部而不是从图像分配 xcom json

 return_value = '{"foo": "bar"\n, "buzz": 2}'
 strtpodbefore = KubernetesPodOperator(namespace='rguonew',
                                image="python:3.6.6-stretch",
                                cmds=["bash", "-cx"],
                                name="fail",
                                task_id="failing-task",
                                arguments=['echo \'{}\' > 
 /airflow/xcom/return.json'.format(return_value)],
                                get_logs= True,
                                xcom_push=True,
                                dag=dag
                                )

所以我尝试了最后的解决方案,做了一个额外的参数,但仍然不起作用,第一个命令没有得到这样的目录返回

strtpodbefore = KubernetesPodOperator(namespace='rguonew',
                                    image="myimages",
                                    name="fail",
                                    image_pull_policy="Always",
                                    cmds=["bash", "-cx"],
                                    arguments=['echo \'{}\' > 
/airflow/xcom/return.json'.format(return_value)],
                                    task_id="failing-task",
                                    get_logs= True,
                                    xcom_push=True,
                                    dag=dag
                                    )

当您设置 xcom_push=True 时,将与您的执行器容器一起创建一个 sidecar 容器。 sidecar 容器将读取执行器的/airflow/xcom/return.json,因此您实际上需要从执行器容器中写入此内容,就像在第二个示例中所做的那样。

看这里:https://stackoverflow.com/a/53568710/10675601 https://stackoverflow.com/a/53568710/10675601

当我们在处理 pod xcom 时,还存在一些与 RBAC 相关的问题需要设置:Airflow k8s 运营商 xcom - 握手状态 403 禁止 https://stackoverflow.com/questions/53827345/airflow-k8s-operator-xcom-handshake-status-403-forbidden

本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)

没有这样的文件或目录 /airflow/xcom/return.json 的相关文章

随机推荐