I am exploring the Hadoop ecosystem and keep encountering Apache Hive. I am confused about its fundamental nature—is it a traditional relational database like MySQL, or is it something else entirely? I understand it uses HiveQL which looks like SQL, but I want to know if it stores data itself or just acts as a layer over HDFS. Could someone clarify the architectural differences and explain the ideal scenarios for using Hive over a standard RDBMS?
3 answers
Apache Hive is not a database in the traditional sense; it is a data warehouse software project built on top of Apache Hadoop. While a standard database like PostgreSQL is designed for online transaction processing (OLTP) and fast row-level updates, Hive is designed for data summarization, query, and analysis of massive datasets (OLAP). Hive does not actually "store" data in its own format. Instead, it provides a logical structure over files stored in HDFS (Hadoop Distributed File System). When you run a HiveQL query, Hive converts that SQL-like code into MapReduce, Tez, or Spark jobs that execute across the cluster. It’s best used for batch processing of petabytes of data rather than real-time application backend support.
Does Hive support ACID transactions like a traditional database? I’ve read that newer versions have added support for inserts and updates, but is it performant enough to be used for frequently changing data? Are you looking to use Hive for a real-time analytics dashboard, or is this primarily for long-running ETL processes where latency isn't a major concern?
Think of Hive as a "SQL-to-MapReduce" translator. It provides the familiarity of SQL to data analysts who don't want to write complex Java code to process data on a Hadoop cluster.
I agree with Barbara. The beauty of Hive is the Metastore. It allows different tools in the Big Data stack to share the same schema definitions for the data stored in HDFS, making it a very powerful central catalog for a data lake.
Jeffrey, that is a critical distinction for anyone coming from a SQL background. To answer your question, while Hive 3.x supports ACID transactions, it is still not built for high-concurrency "point" updates. It uses a delta file mechanism that requires periodic compaction. If Susan's use case involves thousands of updates per second, Hive would be a poor choice. It is much better suited for "Write Once, Read Many" scenarios. For real-time updates in the Hadoop ecosystem, something like Apache HBase or Kudu would be a much more appropriate architectural fit than Hive.