S3 到 Redshift：复制但访问被拒绝

我们以前每天使用 COPY 命令从没有特定策略的存储桶将文件从 s3 复制到 Redshift。

COPY schema.table_staging     
FROM 's3://our-bucket/X/YYYY/MM/DD/'     
CREDENTIALS 'aws_access_key_id=xxxxxx;aws_secret_access_key=xxxxxx'     
CSV     
GZIP     
DELIMITER AS '|'     
TIMEFORMAT 'YYYY-MM-DD HH24:MI:SS';  

由于我们需要提高 S3 存储桶的安全性，因此我们添加了一项策略来授权来自 VPC（我们用于 Redshift 集群的 VPC）或特定 IP 地址的连接。

{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
    {
        "Sid": "DenyAllExcept",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": [
            "arn:aws:s3:::our-bucket/*",
            "arn:aws:s3:::our-bucket"
        ],
        "Condition": {
            "StringNotEqualsIfExists": {
                "aws:SourceVpc": "vpc-123456789"
            },
            "NotIpAddressIfExists": {
                "aws:SourceIp": [
                    "12.35.56.78/32"
                ]
            }
        }
    }
]
}

此策略非常适合使用 AWS CLI 或 boto Python 库从 EC2、EMR 或我们的特定地址访问文件。

这是我们在 Redshift 上遇到的错误：

ERROR: S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid xxxxxx,CanRetry 1
Détail : 
-----------------------------------------------
error:  S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid xxxxxx,CanRetry 1
code:      8001
context:   Listing bucket=our-bucket prefix=X/YYYY/MM/DD/
query:     1587954
location:  s3_utility.cpp:552
process:   padbmaster [pid=21214]
-----------------------------------------------

如果您能在这方面帮助我们，非常感谢，

Damien

ps：这个问题与这个问题非常相似：将数据从 S3 复制到 Redshift - 访问被拒绝

您需要使用 Redshift 的“增强型 VPC 路由”功能。从文档中here:

1. When you use Amazon Redshift Enhanced VPC Routing, Amazon Redshift forces all COPY and UNLOAD traffic between your cluster and your data repositories through your Amazon VPC.

2. If Enhanced VPC Routing is not enabled, Amazon Redshift routes traffic through the Internet, including traffic to other services within the AWS network.

3. For traffic to an Amazon S3 bucket in the same region as your cluster, you can create a VPC endpoint to direct traffic directly to the bucket.