I am currently designing a data pipeline where we receive millions of small JSON files daily. I’ve heard that Hadoop HDFS is notorious for the "Small File Problem" because it overloads the NameNode memory with metadata. Is this still a critical issue in 2024, and what are the best industry-standard practices to consolidate these files before or after they hit the cluster?
3 answers
The small file problem remains a significant bottleneck for NameNode scalability because every object in HDFS (file or block) takes up roughly 150 bytes of metadata in the master node's RAM. When you have millions of tiny files, you'll exhaust memory long before you run out of actual disk space on your DataNodes. In recent deployments, we typically use Apache NiFi to merge records into larger batches (aiming for the 128MB or 256MB block size) before ingestion. Alternatively, using Hadoop Archives (HAR) or SequenceFiles can effectively pack these smaller files into a single container.
Have you considered using a compaction layer like Apache Iceberg or Delta Lake on top of your HDFS to handle these updates and small file merges automatically?
We usually just implement a cron job that runs a MapReduce 'FileConcat' or uses 'CombineFileInputFormat' to group those small splits into a single mapper for better performance.
I agree with Emily; 'CombineFileInputFormat' is a lifesaver for MapReduce jobs. It prevents the overhead of launching one mapper per tiny file, which usually kills the job's execution time.
Robert, that's a great point. While Iceberg helps with table-level compaction, it still requires a periodic maintenance job to rewrite those small files into larger parity files. For Michael's specific JSON ingestion, I'd suggest a 'Lambda' approach where you land data in a 'staging' area first, run a Spark job to coalesce the partitions, and then move the "clean" larger files into the main HDFS production directory to keep the NameNode healthy.