I am working on a Software Development project that involves auditing our AWS S3 storage. Since S3 is a flat storage system and doesn't use real folders, I am struggling to write a script that displays the hierarchy of "directories" and "files" correctly. How can I use the Boto3 library to iterate through a bucket and list both the prefixes and the object keys? Specifically, I need to know the difference between using the list_objects_v2 method versus the S3 Resource approach for large buckets.
3 answers
The most efficient way to do this in a Software Development environment is using the list_objects_v2 method with the Delimiter parameter. Even though S3 is flat, setting Delimiter='/' tells AWS to treat the forward slash as a folder separator. You can then access "folders" via CommonPrefixes and "files" via Contents.
If a bucket has more than 1,000 objects, will this script fail to show the rest of the files, or does Boto3 handle pagination automatically?
You can also use the high-level s3.Bucket.objects.all() method if you just want a flat list of every single file without worrying about folders.
I agree with David. If you don't need to distinguish between directories and files for your Software Development task, the Resource API is much more "Pythonic" and easier to read than the Client API.
Michael, that's a crucial point! list_objects_v2 only returns up to 1,000 items per call. In professional Software Development, you should use a Paginator. It handles the NextContinuationToken for you so you don't miss any data. I switched my Cloud Technology scripts to use paginator = s3.get_paginator('list_objects_v2'), which allows me to loop through millions of objects without the script cutting off prematurely.