I've spent 10 years as a SQL developer and now I'm looking to move into Big Data Engineering. Everyone says Hadoop is built on Java, and I'm worried my lack of coding experience will be a barrier. Can I survive in a Hadoop environment using just Hive and Impala, or is MapReduce/Spark coding in Java/Scala mandatory for senior roles?
3 answers
The good news is that about 80% of Big Data work in the industry is now done via SQL abstractions like Apache Hive, Spark SQL, or Presto. Your SQL skills are incredibly valuable. However, to be a "Senior" Engineer, you will eventually need to understand the underlying framework. You don't need to be a Java guru, but you should be able to read Java code to debug why a specific User Defined Function (UDF) is failing or to tune your memory settings. I'd suggest learning Python first; it's easier than Java and is the primary language for PySpark, which is very popular now.
Are you finding that most job descriptions in your area are asking specifically for Hadoop MapReduce, or are they pivoting more toward the newer Snowflake and Databricks stacks?
Just focus on HiveQL and Spark SQL. I've been a data engineer for 4 years and I've rarely had to write a raw MapReduce job in Java. It's almost all SQL now.
That's very encouraging, Nancy! I'll start with Hive and then look into PySpark since Python seems much more approachable than trying to learn Java from scratch.
Daniel, I'm seeing a mix. The big banks and insurance companies still have massive on-prem Hadoop clusters and need people who understand Hive and HDFS security. But tech startups are definitely on the cloud-native side. Lisa, my advice is to master Hive first because the syntax is so close to SQL. Then, learn how Hive translates your SQL into MapReduce or Tez jobs. Understanding that "behind the scenes" logic is what separates a developer from an architect.