I've used Apache Spark for years, but lately, I see everyone raving about DuckDB for smaller, in-memory processing. Is it worth keeping Spark on my resume for 2024, or is the industry moving toward lighter, faster tools that don't require a whole cluster to run a simple SQL query? I'm worried about my skills becoming obsolete.
3 answers
Spark is definitely not obsolete! While DuckDB is incredible for "small data" (anything that fits on your laptop), Spark is still the only real answer for processing petabytes of data across a distributed cluster. Think of it this way: DuckDB is a Ferrari for the city, but Spark is a freight train for the country. In my role at a major streaming service, we use DuckDB for quick local testing and EDA, but our production pipelines that handle billions of events per hour are still 100% Spark. Keep both on your resume; they show you know how to choose the right tool for the scale.
Stephanie, what about the cost? We found that running Spark clusters in AWS was eating our budget alive. Have you seen people moving to tools like Polars to save on cloud costs?
I love DuckDB for my local development. It makes my "Inner Loop" so much faster. But yeah, Spark is still the heavy hitter for the real work.
Ryan, exactly! Using DuckDB for dev and Spark for prod is a very common and efficient "hybrid" workflow these days.
Kevin, cloud cost optimization is a huge part of the job now. We actually use Polars for our medium-sized ETL jobs (up to 100GB) because it’s much faster and cheaper than spinning up a Spark cluster for something that can run on a single high-memory node. The trick is knowing the "cutoff point" where the overhead of Spark becomes worth it. Being a "cost-aware" engineer is a major selling point in the current market.