Everyone talks about AI and Data Science, but I’ve noticed that without solid pipelines, those models are useless. Is Data Engineering actually the most underrated role right now? It seems like there are more job openings for engineers to build the "plumbing" than there are for scientists to build the models. Is this the better long-term bet for career stability?
3 answers
It is definitely underrated in terms of public perception, but not in terms of salary. In late 2023, my firm shifted its budget to hire three data engineers for every one data scientist. The reality is that companies are drowning in "dirty data" and they’ve realized they can't do fancy analytics until the data is cleaned, moved, and stored correctly. From a stability standpoint, an engineer who knows Spark, Airflow, and Snowflake is almost recession-proof because they manage the literal lifeblood of the company’s decision-making process.
Megan, do you think the barrier to entry is higher though? It feels like you need to be a better "software engineer" to succeed in data engineering compared to just knowing some Python and Statistics for a data science role.
I switched from backend dev to data engineering last year and my salary jumped 25%. The demand is massive and the work is much more rewarding.
That’s a common story, Brian. The transition from backend to data is very natural since the core principles of building robust systems are identical.
Gregory, you hit the nail on the head. You need to understand systems, networking, and distributed computing. It’s definitely a "heavier" technical lift, but that’s exactly why it’s so well-paid and why there’s such a talent shortage. If you can bridge the gap between software development and data analysis, you become indispensable to any enterprise looking to scale their AI efforts.