I am a web developer transitioning into Big Data. I’m used to working with PostgreSQL and MySQL. Why should I invest time in learning HDFS architecture? Doesn't a distributed SQL database offer the same scalability without the complexity of managing a name node and data nodes?
3 answers
The core difference lies in the "Schema-on-Read" vs "Schema-on-Write" philosophy. In an RDBMS, you must define your schema before you can store data, which is slow for unstructured logs or sensor data. HDFS allows you to dump raw data (JSON, CSV, Parquet) immediately and figure out the structure later when you query it. More importantly, HDFS scales horizontally on "commodity hardware." If you need more storage, you just add another $5k server. Scaling an RDBMS to petabytes usually requires very expensive, specialized hardware. For "Write-Once, Read-Many" workloads involving terabytes of data, HDFS is significantly more cost-effective and performs better for massive parallel processing tasks like those found in Data Science.
That makes sense for storage, but isn't the "latency" of HDFS a major drawback? If I need to fetch a single record by its ID, won't a PostgreSQL index always beat a Hadoop scan?
HDFS is designed for high-volume data streams. If your data fits on a single server, stick to SQL. If it doesn't, Hadoop is your only real choice.
Sarah's advice is the best rule of thumb. Don't add the complexity of a distributed cluster unless the volume of your data literally forces you to do so!
Comment: Anthony, you are 100% correct. HDFS is a throughput-oriented system, not a latency-oriented one. It's built for "Batch Processing." If you need sub-second point lookups, you should use Apache HBase, which is a NoSQL database built on top of HDFS. It gives you the best of both worlds: the massive storage of Hadoop with the indexed speed of a traditional database.