I need to download a specific "folder" (prefix) from my S3 bucket that contains several hundred objects and subfolders. The AWS Console only seems to allow downloading one file at a time, which is not feasible for my project. Is there a command-line method or a script that can recursively pull the entire directory structure while maintaining the file hierarchy on my local drive?
3 answers
The most efficient way to handle this is using the AWS CLI. If you have the CLI configured, you can use the sync command, which is generally better than cp for large directories because it only downloads new or updated files. Run: aws s3 sync s3://your-bucket-name/folder-name/ ./local-destination/. This command will replicate the entire S3 prefix structure onto your local machine. If you prefer a one-time "blind" copy without checking for existing files, you can use aws s3 cp s3://your-bucket-name/folder-name/ ./local-destination/ --recursive. Both methods are industry standards for DevOps and Cloud Engineers because they are significantly faster than using the web-based console.
Does the sync command handle deletions on the local side as well? I'm worried that if I remove a file from my S3 bucket, it might still linger in my local folder after I run the command again.
For those who prefer Python, you can use a Boto3 script to loop through all objects with a specific prefix and call download_file for each. It gives you more control over error handling.
I agree with David. While the CLI is great for manual tasks, Boto3 is the way to go if you're building this into a larger automated pipeline. Just be sure to create the local directories dynamically using os.makedirs() as you loop through the keys, or the download will fail when it hits a subfolder!
That’s a sharp observation, Michael! By default, aws s3 sync does not delete local files that are missing from the source. If you want your local folder to be an exact mirror of the bucket, you need to add the --delete flag to your command. This ensures that any file no longer present in S3 is also purged from your local directory, keeping your data clean and synchronized.