I am working on an automation script to organize my AWS S3 bucket data. I've successfully created buckets, but I can't find a create_folder method in the Boto3 documentation. I know that S3 is a flat storage system and doesn't use real folders, but I need the directory structure to appear correctly in the AWS Management Console for my team. What is the standard way to create these "keys" using put_object, and do I need to include a specific suffix to make it behave like a directory?
3 answers
Since Amazon S3 is a flat object store, "folders" are actually just empty objects with a key that ends in a forward slash (/). To create one using Boto3, you use the put_object method. You simply specify the bucket name and a key that ends in a slash, such as Key='my-folder-name/'. You don't need to provide a Body parameter, as the object itself is essentially empty. Once this is executed, the AWS Console will interpret that trailing slash as a directory delimiter and display it as a folder. However, keep in mind that if you upload an object directly to my-folder-name/file.txt, S3 will automatically show the folder in the UI even if you never explicitly created the empty folder object first.
The most efficient way is to just upload your file with the full path. S3 creates the visual hierarchy automatically, which saves you from making unnecessary API calls to create empty folders
If I create a folder object using put_object with a trailing slash, will deleting all the files inside that folder also delete the folder itself, or will the empty "folder" object persist in my bucket?
Brandon, if you explicitly created the folder using a put_object call (creating a 0-byte object), it will persist even after you delete all other files because it is its own unique object in the metadata. However, if the folder was only visible because of the path naming of your files (like uploads/photo.jpg), then deleting the files will make the folder "disappear" from the console. For SEO and organizational consistency, I usually recommend creating the explicit folder object if you want to maintain a specific directory structure for users who manually navigate the AWS UI.
I agree with Jennifer. I stopped creating empty folder objects months ago. Just naming your keys like year/month/day/file.csv is much more efficient for high-volume data science pipelines, and Boto3 handles the pathing perfectly.