We recently added a rack of servers with NVMe SSDs to our existing HDD-based Hadoop cluster. How do I tell HDFS to store "hot" tables on the SSDs for faster Hive queries while keeping the "cold" historical data on the cheaper HDDs? Is this handled automatically, or do I need to tag the DataNodes and set specific policies?
3 answers
HDFS Heterogeneous Storage is a powerful tool for this. First, you must label your DataNode storage directories in hdfs-site.xml as either [SSD] or [DISK]. Once the NameNode is aware of the storage types, you can use Storage Policies. There are several built-in: ALL_SSD (for maximum speed), ONE_SSD (one replica on SSD, others on HDD), and WARM (mostly HDD). You apply these to specific folders using the command hdfs storagepolicies -setStoragePolicy -path <path> -policy <policy>. HDFS will then try its best to place new blocks on the requested hardware. This is a game-changer for frequently queried "fact tables" in Hive, as moving them to SSDs can reduce query times by 5x-10x without the cost of making the entire cluster SSD-based.
This sounds amazing, but what happens if the SSD storage is full? Does the write fail, or does HDFS have a "fallback" mechanism to write the data to the regular HDDs until we clear some space on the NVMe drives?
You can even use LAZY_PERSIST for temporary files, which writes to RAM first and then to disk, providing even faster speeds than SSD for transient data.
I’ve used LAZY_PERSIST for some Spark intermediate files and the performance boost was incredible. Just be careful as it carries a higher risk of data loss if the node crashes before the RAM is flushed to disk.
Anthony, HDFS is quite smart about that. If you use the ALL_SSD policy and the SSDs are full, it will automatically fall back to DISK to ensure the write doesn't fail. However, you should run the "Mover" tool periodically. The Mover is like the Balancer, but it specifically moves blocks to the correct "tier" if they were written to the wrong hardware due to a fallback event.