使用以下代码将大文件从 S3 (>5GB) 读取到 lambda 中:
import json
import boto3
s3 = boto3.client('s3')
def lambda_handler(event, context):
response = s3.get_object(
Bucket="my-bucket",
Key="my-key"
)
text_bytes = response['Body'].read()
...
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
但是我收到以下错误:
"errorMessage": "signed integer is greater than maximum"
"errorType": "OverflowError"
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 13, in lambda_handler\n text_bytes = response['Body'].read()\n"
" File \"/var/runtime/botocore/response.py\", line 77, in read\n chunk = self._raw_stream.read(amt)\n"
" File \"/var/runtime/urllib3/response.py\", line 515, in read\n data = self._fp.read() if not fp_closed else b\"\"\n"
" File \"/var/lang/lib/python3.8/http/client.py\", line 472, in read\n s = self._safe_read(self.length)\n"
" File \"/var/lang/lib/python3.8/http/client.py\", line 613, in _safe_read\n data = self.fp.read(amt)\n"
" File \"/var/lang/lib/python3.8/socket.py\", line 669, in readinto\n return self._sock.recv_into(b)\n"
" File \"/var/lang/lib/python3.8/ssl.py\", line 1241, in recv_into\n return self.read(nbytes, buffer)\n"
" File \"/var/lang/lib/python3.8/ssl.py\", line 1099, in read\n return self._sslobj.read(len, buffer)\n"
]
我使用的是 Python 3.8,我在这里发现了 Python 3.8/9 的一个问题,这可能是原因:https://bugs.python.org/issue42853
有没有办法解决?