使用特定 SA 通过 Cloud Build 部署 Dataflow 管道

我竭尽全力尝试使用特定 SA（而不是默认的 Cloud Build SA）从 Cloud Build 部署数据流管道，但到目前为止尚未成功。

我遵循了这个过程 -https://cloud.google.com/build/docs/secure-builds/configure-user-specified-service-accounts https://cloud.google.com/build/docs/securing-builds/configure-user-specified-service-accounts

该密钥被授予以下角色（比文档推荐的角色还要多）

• 数据流管理
• 数据流工作者
• 日志编写器
• 服务帐户用户
• 存储管理员
• 存储对象管理

命令行部署

os.system("python3.7 /workspace/runner.py \
    --runner=DataflowRunner \
    --job_name=test \
    --project=XXXXXX \
    --temp_location=gs://staging/tmp \
    --region=europe-west1 \
    --environment=dev \
    --max_num_workers=20 \
    --autoscaling_algorithm=THROUGHPUT_BASED \
    --setup_file=/workspace/setup.py \
    --servic[email protected] /cdn-cgi/l/email-protection \
    ")

Error:

apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/my-prj/locations/europe-west1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 18 Apr 2023 14:03:30 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '403', 'content-length': '812', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 403,
    "message": "(3b12042024f17c98): Current user cannot act as service account [email protected] /cdn-cgi/l/email-protection. Please grant your user account one of [Owner, Editor, Service Account Actor] roles, or any other role that includes the iam.serviceAccounts.actAs permission. See https://cloud.google.com/iam/docs/service-accounts-actas for additional details. Causes: (3b12042024f17239): Current user cannot act as service account [email protected] /cdn-cgi/l/email-protection. Please grant your user account one of [Owner, Editor, Service Account Actor] roles, or any other role that includes the iam.serviceAccounts.actAs permission. See https://cloud.google.com/iam/docs/service-accounts-actas for additional details.",
    "status": "PERMISSION_DENIED"
  }
}

这是我单击链接时 Dataflow 的输出消息HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/my-prj/locations/europe-west1/jobs?alt=json>

{
  "error": {
    "code": 401,
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "status": "UNAUTHENTICATED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "CREDENTIALS_MISSING",
        "domain": "googleapis.com",
        "metadata": {
          "service": "dataflow.googleapis.com",
          "method": "google.dataflow.v1beta3.JobsV1Beta3.ListJobs"
        }
      }
    ]
  }
}

默认情况下，如果您未在启动的命令行中设置服务帐户Dataflow工作，默认Compute使用服务帐户。

在启动的命令行中Dataflow作业，您可以设置用户指定的服务帐户 https://cloud.google.com/dataflow/docs/concepts/security-and-permissions :

Java 和 Maven 的示例：

mvn compile exec:java \
  -Dexec.mainClass=com.package.MainClass \
  -Dexec.args=" \
  --project=project \
  --runner=DataflowRunner \
  --serviceAccount=your-sa-email
  "
  ....

使用 Python 的示例：

python -m folder.main \
    --project=project \
    --runner=DataflowRunner \
    --region=europe-west1 \
    --setup_file=./setup.py \
    --temp_location=gs://mazlum_dev/dataflow/temp \
    --service_account_email=your-sa-email
  ....

该用户指定的服务帐户需要具有启动作业所需的角色以及允许充当以下角色的角色：

• 数据流开发人员
• 数据流工作者
• 服务帐户令牌创建者