We are running out of disk space on our Hadoop 3.x cluster and I am looking into Erasure Coding as an alternative to the standard 3x replication. However, I am worried about the impact on read performance. Is it possible to apply Erasure Coding only to specific directories with "cold data" while keeping our active datasets replicated for speed? What are the best CLI commands to manage these policies without downtime?
3 answers
HDFS Erasure Coding (EC) is a fantastic way to save up to 50% of your storage space by moving away from the 3x replication factor. To answer your question, yes, EC is applied at the directory level via storage policies. You can use the hdfs ec -setPolicy -path <path> -policy <policyName> command to target specific "cold" folders. I recommend using the RS-6-3-1024k policy for large files as it offers a great balance between storage savings and recovery overhead. Keep in mind that EC is significantly more CPU-intensive during the "read-on-failure" scenarios because the data must be reconstructed from parity shards. For your active, frequently accessed "hot data," sticking with replication is definitely the safer bet to avoid latency spikes during high-concurrency MapReduce or Spark jobs.
That is a great technical summary, Deborah. But what happens to the existing data in a folder when I apply an Erasure Coding policy? Does Hadoop automatically convert the old replicated blocks into the new EC format, or do I need to manually rewrite the files to trigger the change?
Be careful with small files when using Erasure Coding. EC works best on large files; if your files are smaller than the cell size, you actually won't see much storage benefit and might even increase metadata overhead.
Patricia is right. We tried EC on a log directory with tiny files and it was a disaster. Stick to large archive files for EC to get the best results.
James, that’s a common point of confusion. Setting a policy only affects new files written to that directory. For existing data, you’ll need to use the distcp command to copy the data to a new folder or move it out and back in. Only then will the NameNode instruct the DataNodes to stripe the data and calculate the parity according to your new Erasure Coding settings.