python - Any way to write files DIRECTLY to S3 using boto3?

Question

I wrote a python script to process very large files (few TB in total), which I'll run on an EC2 instance. Afterwards, I want to store the processed files in an S3 bucket. Currently, my script first saves the data to disk and then uploads it to S3. Unfortunately, this will be quite costly given the extra time spent waiting for the instance to first write to disk and then upload.

Is there any way to use boto3 to write files directly to an S3 bucket?

Edit: to clarify my question, I'm asking if I have an object in memory, writing that object directly to S3 without first saving the object onto disk.

Answer 1

You can use put_object for this. Just pass in your file object as body.

For example:

import boto3

client = boto3.client('s3')
response = client.put_object( 
    Bucket='your-s3-bucket-name',
    Body='bytes or seekable file-like object',
    Key='Object key for which the PUT operation was initiated'
)