I’m curious about the underlying storage layer for Big Data. Traditionally, HDFS (Hadoop Distributed File System) was the king because of "Data Locality," where the compute happened on the same node as the storage. But now, everyone is moving to S3 or Azure Blob Storage. For a company handling 100 petabytes, is the loss of data locality a major performance hit, or has modern networking speed made the "HDFS vs Object Storage" debate irrelevant?
3 answers
Do you think the move to object storage makes it harder to manage "Small File" problems that typically plague HDFS users?
The "Data Locality" argument for HDFS has largely faded because of 100Gbps networking in modern data centers. While HDFS is technically faster for certain specific heavy-compute tasks, the operational cost of managing thousands of physical servers is massive. Object storage (S3) separates compute from storage, allowing you to turn off your expensive Spark clusters when you aren't using them while keeping your data safely in S3 for pennies. At the 100PB scale, the durability and "Infinite" scalability of S3 far outweigh the marginal performance gains you might get from an HDFS cluster.
HDFS is still great for on-premise "Air-gapped" environments where you can't use the cloud, but for 99% of businesses, S3 has clearly won the storage war.
Agreed, Dorothy. The flexibility to scale compute independently of storage is just too good of a financial incentive for companies to ignore.
Richard, the "Small File" problem is actually worse in object storage because each metadata request (GET/LIST) has a cost and latency. In HDFS, the NameNode handles this, but it can become a bottleneck. In S3, you really have to use file formats like Parquet or Avro and ensure you are "compacting" your data into 128MB or 256MB chunks. If you have millions of 1KB files in S3, your query performance will be abysmal. We have an automated "Compaction Job" that runs every night to fix this exact issue.