amazon web services - downloading a file from Internet into S3 bucket

Question

I would like to grab a file straight of the Internet and stick it into an S3 bucket to then copy it over to a PIG cluster. Due to the size of the file and my not so good internet connection downloading the file first onto my PC and then uploading it to Amazon might not be an option.

Is there any way I could go about grabbing a file of the internet and sticking it directly into S3?

Answer 1

Download the data via curl and pipe the contents straight to S3. The data is streamed directly to S3 and not stored locally, avoiding any memory issues.

curl "https://download-link-address/" | aws s3 cp - s3://aws-bucket/data-file

As suggested above, if download speed is too slow on your local computer, launch an EC2 instance, ssh in and execute the above command there.