I need to pull several gigabytes of data from a private Amazon S3 bucket for a local audit. I have the AWS CLI configured, but I’m confused about whether I should use the cp or sync command. What is the standard way to download an entire folder recursively while ensuring I don't miss any subdirectories or hidden files? Also, are there any flags to speed up the transfer for large datasets?
3 answers
The most straightforward way to download a folder is using the aws s3 cp command with the --recursive flag. The syntax would be aws s3 cp s3://your-bucket-name/folder/ ./local-dir/ --recursive. However, if you are dealing with a massive dataset and want to avoid re-downloading files you already have, I highly recommend using aws s3 sync. The sync command compares the source and destination and only downloads new or updated files, which is a lifesaver for large-scale cloud migrations or backups. To speed things up, you can increase the max_concurrent_requests in your AWS configuration to leverage more threads during the transfer process.
Does the sync command automatically delete files in my local folder if they were removed from the S3 bucket, or do I need a specific flag for that?
For a single file, just use aws s3 cp s3://bucket/file.txt .. For folders, always add --recursive. If the bucket is huge, use sync to save bandwidth and time.
I agree with Jennifer. I’d also add that using the --exclude "*" and --include "*.jpg" flags is great if you only need specific file types from a cluttered bucket.
David, the sync command does not delete local files by default. If you want your local folder to be an exact mirror of the S3 bucket, you must add the --delete flag. Be very careful with this, though, as it will permanently remove any local files that aren't present in the source S3 path. It’s usually best to run a --dryrun first to see exactly what would be deleted before committing to the actual command in a production environment.