从 ASP.NET Core Web API 将多 GB 文件流式传输到 AWS S3

我希望通过 ASP.NET Core Web API 在 AWS S3 存储桶中创建一个大型（多 GB）文件。该文件足够大，我不想加载Stream在将其上传到 AWS S3 之前先将其存储到内存中。

Using PutObjectAsync()我被迫预先填充Stream在将其传递给 AWS SDK 之前，如下所示：

var putObjectRequest = new PutObjectRequest
{
    BucketName = "my-s3-bucket",
    Key = "my-file-name.txt",
    InputStream = stream
};
var putObjectResponse = await amazonS3Client.PutObjectAsync(putObjectRequest);

我的理想模式是 AWS SDK 返回一个StreamWriter（某种意义上）我可以Write()多次然后Finalise()当我完成时。

关于我的挑战的两个问题：

• 我是否对必须预先填充有误解Stream在致电之前PutObjectAsync()?
• 我应该如何上传大（多 GB）文件？

对于这样的情况AWS docs http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html提供两个选项：

• 使用 AWS .NET 开发工具包进行分段上传（高级 API） http://docs.aws.amazon.com/AmazonS3/latest/dev/usingHLmpuDotNet.html
• 使用 AWS .NET 开发工具包进行分段上传（低级 API） http://docs.aws.amazon.com/AmazonS3/latest/dev/usingLLmpuDotNet.html

高级 API 只是建议您创建一个TransferUtilityUploadRequest http://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html with a PartSize指定，这样类本身就可以上传文件，而不需要自己维护上传。在这种情况下，您可以通过订阅来获取分段上传的进度StreamTransferProgress http://docs.aws.amazon.com/AmazonS3/latest/dev/LLTrackProgressMPUNet.html事件。您可以上传文件、流或目录。

低级API http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFileDotNet.html显然，更复杂，但更灵活 - 您可以启动上传，然后循环上传文件的下一部分。文档中的示例代码：

var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);

// List to store upload part responses.
var uploadResponses = new List<UploadPartResponse>();

// 1. Initialize.
var initiateRequest = new InitiateMultipartUploadRequest
    {
        BucketName = existingBucketName,
        Key = keyName
    };

var initResponse = s3Client.InitiateMultipartUpload(initRequest);

// 2. Upload Parts.
var contentLength = new FileInfo(filePath).Length;
var partSize = 5242880; // 5 MB

try
{
    long filePosition = 0;
    for (var i = 1; filePosition < contentLength; ++i)
    {
        // Create request to upload a part.
        var uploadRequest = new UploadPartRequest
            {
                BucketName = existingBucketName,
                Key = keyName,
                UploadId = initResponse.UploadId,
                PartNumber = i,
                PartSize = partSize,
                FilePosition = filePosition,
                FilePath = filePath
            };

       // Upload part and add response to our list.
       uploadResponses.Add(s3Client.UploadPart(uploadRequest));

       filePosition += partSize;
   }

   // Step 3: complete.
   var completeRequest = new CompleteMultipartUploadRequest
       {
           BucketName = existingBucketName,
           Key = keyName,
           UploadId = initResponse.UploadId,
        };

   // add ETags for uploaded files
   completeRequest.AddPartETags(uploadResponses);

   var completeUploadResponse = s3Client.CompleteMultipartUpload(completeRequest);     
}
catch (Exception exception)
{
    Console.WriteLine("Exception occurred: {0}", exception.ToString());
    var abortMPURequest = new AbortMultipartUploadRequest
        {
            BucketName = existingBucketName,
            Key = keyName,
            UploadId = initResponse.UploadId
        };
    s3Client.AbortMultipartUpload(abortMPURequest);
}

异步版本UploadPart http://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MS3S3UploadPartAsyncUploadPartRequestCancellationToken.html也可用，因此如果您需要完全控制上传，则应该调查该路径。