Our data lake is growing to petabytes, and our storage costs are out of control. Is S3 Intelligent-Tiering the best way to handle data with unknown access patterns, or should we manually move files to Glacier? I'm worried about the monitoring fees for small files making the "savings" vanish.
3 answers
For petabyte-scale data with unknown access, Intelligent-Tiering is usually the best bet because it automates the move between Frequent, Infrequent, and Archive Instant Access tiers without retrieval fees. However, your concern about small files is valid; there is a monthly monitoring and automation fee per object. If your dataset consists of millions of 10KB files, that fee can indeed eat up your savings. A better strategy for small files is to aggregate them into larger ZIP or Tar archives before uploading to S3, which reduces the object count and maximizes the benefit of the tiering logic.
Susan’s aggregation tip is essential! But what about the "Deep Archive" tier? Intelligent-Tiering only goes as far as "Archive Instant Access" by default. If you have data you know you won't touch for years, shouldn't you still use S3 Lifecycle policies to push it to Glacier Deep Archive for the lowest possible cost?
Use S3 Storage Lens to find your biggest "waste" areas first. It gives you a great dashboard to see where you have non-current versions or incomplete multipart uploads taking up space.
Spot on, Karen. We found 50TB of "Incomplete Multipart Uploads" using Storage Lens. Deleting those gave us immediate savings without having to change our storage classes at all!
Joseph, you can actually opt-in to the "Deep Archive" tier within Intelligent-Tiering now. It takes longer to retrieve, but it gives you that rock-bottom pricing automatically if the data hasn't been touched for 180 days. It combines the best of both worlds—automation and deep-freeze pricing.