使用 JAVA 在 S3 存储桶上放置/读取文件

我试图将文件放在 S3 上并使用 JAVA 从 S3 存储桶读取文件，但遇到无法执行 HTTP 请求。

当我尝试列出存储桶时，它工作正常。

对我来说，似乎只有 listBuckets() 方法在工作，而所有其他方法（如 putObject()、listObjects()、createBucket() 等）都抛出相同的错误无法执行 HTTP 请求

这是我正在努力修复的代码，我使用了很多方法，但除了 listBuckets() 之外没有一个有效 -

package test;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;

 public class JavaS3Conn {

//private static Logger log = Logger.getLogger(JavaS3Conn.class);

public static void main(String[] args) {

    String access_key = "access_key";
    String secret_key = "secret_key";
    String end_point = "http://end_point:port/";
    String bucketName = "bucketName";
    String stringObjKeyName = "Hello_S3.txt";
    String fileObjKeyName = "Hello_S3.txt";
    String fileName = "C:\\Users\\Desktop\\hello_S3.txt";
    try {
    BasicAWSCredentials creds = new BasicAWSCredentials(access_key, secret_key); 
ClientConfiguration clientConfig = new ClientConfiguration()
                .withProxyHost("wdctestlab-ecs1-node1.systems.uk.hsbc").withProxyPort(9020)
                .withNonProxyHosts("");
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withClientConfiguration(clientConfig).withCredentials(new AWSStaticCredentialsProvider(creds)).withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(end_point,Regions.DEFAULT_REGION.getName())).build();
    /*
    if(s3Client.doesBucketExistV2(bucketName)) {
        System.out.println("Bucket name is not available."
          + " Try again with a different Bucket name.");
        return;
    }*/


    //s3Client.createBucket(bucketName);

    ObjectListing objectListing = s3Client.listObjects(bucketName);
    for(S3ObjectSummary os : objectListing.getObjectSummaries()) {
        System.out.println(os.getKey());
    }
    /*
    s3Client.putObject(
              bucketName, 
              stringObjKeyName, 
              "Uploaded String Object"
            );
    */
    PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName));
    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentType("plain/text");
    metadata.addUserMetadata("x-amz-meta-title", "someTitle");
    request.setMetadata(metadata);
    //s3Client.putObject(request);
    //s3Client.listObjects(bucketName);

    for (Bucket bucket : s3Client.listBuckets()) {
        System.out.println(" - " + bucket.getName());
    }

    } catch (AmazonServiceException e) {
        e.printStackTrace();
    } catch (SdkClientException e) {
        e.printStackTrace();            

    }

}

}

错误 -

 com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket.endpoint
    at 
 com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175 
     )
         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4854)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:880)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:848)
    at test.JavaS3Conn.main(JavaS3Conn.java:51)
Caused by: java.net.UnknownHostException: bucket.endpoint
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
    at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
    at com.amazonaws.http.conn.$Proxy3.connect(Unknown Source)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)

看来我必须添加客户端配置来解决无法执行 HTTP 的问题

ClientConfiguration clientConfig = new ClientConfiguration()
                .withProxyHost("wdctestlab-ecs1-node1.systems.uk.hsbc").withProxyPort(9020)
                .withNonProxyHosts("");

我在 Mac 上的 localstack（devlocal s3 模拟器）docker 容器上运行 s3 并收到此错误（“SdkClientException：无法执行 HTTP 请求：bucketname.localhost”、“UnknownHostException”）。并且 listBuckets 有效。

将其放入 /etc/hosts 中为我解决了这个问题：

127.0.0.1       bucketname.localhost

以下替代解决方案也为我修复了该问题，但此路径样式访问不适用于 2020 年 9 月之后创建的真实 s3 存储桶（而不是 localstack s3 存储桶）：

.withPathStyleAccessEnabled(true)

（添加到 AmazonS3ClientBuilder 设置行）

Why:

请注意bucket.endpoint 上的UnknownHostException - 为什么它尝试将存储桶放入DNS 名称中？：

com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket.endpoint
...
Caused by: java.net.UnknownHostException: bucket.endpoint

对我来说，它是bucketname.localhost，因为我试图连接到在本地主机上运行的localstack S3端点。

新的 S3 存储桶命名使用虚拟托管样式命名，其中存储桶位于路径的前面。bucketname.s3.amazonaws.com not s3.amazonaws.com/bucketname

虚拟托管风格是默认的。

AWS 已弃用路径样式命名。 2020 年 9 月之后创建的所有存储桶都不能使用路径样式。更多信息：https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story

localstack 已经实现了对虚拟主机样式的支持，但如果您在 Docker 容器中本地运行它，则必须为每个存储桶名称创建主机别名。https://github.com/localstack/localstack/issues/2631 https://github.com/localstack/localstack/issues/2631

当在 linux 中运行 docker 时（并使用 systemd-resolved - Fedora/Ubuntu ），它可以工作。我认为这是因为以下行在 Linux 上有效，但在 Mac 上无效：ping abc.localhost（其中 abc 可以是任何东西）。这称为 localhost 子域。

对于 Mac（或不使用 systemd-resolved 的 Linux），另一个选项是将行添加到 /etc/hosts 或使用brew install dnsmasq（dnsmasq 也在 Linux 上）-https://serverfault.com/questions/118378/in-my-etc-hosts-file-on-linux-osx-how-do-i-do-a-wildcard-subdomain#118589 https://serverfault.com/questions/118378/in-my-etc-hosts-file-on-linux-osx-how-do-i-do-a-wildcard-subdomain#118589.

Linux systemd 解析的 localhost-subdomain 信息来源：https://unix.stackexchange.com/questions/401966/how-does-every-subdomain-of-localhost-point-to-localhost-on-fedora https://unix.stackexchange.com/questions/401966/how-does-every-subdomain-of-localhost-point-to-localhost-on-fedora

我跟踪了此点的虚拟主机主机名：（sdk 版本 1.11）

AmazonS3Client.doesObjectExist
 getObjectMetadata
  createRequest
   resolveRequestEndpoint
    new S3RequestEndpointResolver(..)

S3RequestEndpointResolver.resolveRequestEndpoint
 shouldUseVirtualAddressing
  BucketNameUtils.isDNSBucketName
   isValidV2BucketName
    isValidV2BucketName
 request.setEndpoint(convertToVirtualHostEndpoint(endpoint, bucketName));