I am working on a Cloud Technology project and need to delete a specific "folder" within an S3 bucket using a Python script. Since S3 is an object store and not a file system, I am struggling with the Boto3 syntax to remove a prefix and all the objects under it. Is there a way to perform a bulk delete, or do I have to iterate through every single object key manually to clear the directory?
3 answers
To delete a folder in S3 via Boto3, you must remember that S3 is a flat structure. A "folder" is just a zero-byte object ending in a forward slash or simply a prefix shared by other objects. To delete it, you first need to list all objects with that prefix using list_objects_v2. Once you have the list of keys, you can use the delete_objects method to remove up to 1,000 objects in a single batch request. This is the most efficient way to handle cleanup in Software Development and Cloud Technology workflows. If you have more than 1,000 objects, you will need to loop through the pagination.
Does your S3 bucket have versioning enabled? If it does, simply deleting the object won't actually free up space; it will just create a delete marker, leaving the old versions intact in the background.
The easiest way for small folders is using the high-level s3.Bucket.objects.filter(Prefix='folder_name/').delete() method. It handles the batching logic for you automatically behind the scenes.
I agree with Linda. The resource-level API in Boto3 is much more "Pythonic" and saves you from writing the manual loop for the delete_objects batching, which is perfect for most Cloud Technology tasks.
Michael, thanks for bringing that up! I checked, and versioning is indeed enabled on our production bucket. To truly "delete" the folder, I found I had to iterate through the object_versions instead of just the objects. My script now targets version_id to ensure those hidden files aren't hanging around and inflating our AWS bill. It's a crucial distinction for anyone working in Cloud Technology to understand before they start writing cleanup scripts.