I’ve seen a lot of talk about high-performance Data Engineering tools being rewritten in Rust. While Python is great for productivity, it’s slow for heavy data processing. Should I start learning Rust to stay competitive, or is the Python ecosystem (with libraries like Polars and Pydantic) enough to keep me relevant for the next few years in this fast-moving domain?
3 answers
Python is far from dead in Data Engineering, mainly because of its massive ecosystem and ease of use for data scientists. However, the "internals" are definitely shifting. Libraries like Polars are written in Rust but provide a Python API, giving you the best of both worlds. Unless you want to build the tools themselves—like the next big database or orchestration engine—you don't strictly need to be a Rust expert. That said, understanding how Rust handles memory and concurrency can make you a much better Data Engineering professional because it changes how you think about data structures and performance bottlenecks in your existing Python scripts.
Have you noticed any specific Data Engineering tasks where Python's GIL (Global Interpreter Lock) is causing you actual production delays?
I think the future of Data Engineering is "Python on the outside, Rust on the inside." You get the speed of a low-level language without the complexity.
Brandon is exactly right. The "abstraction" layer is staying Pythonic, which is great for Data Engineering productivity, while the engine under the hood gets a major speed boost.
Honestly, Steven, it’s mostly when we are doing heavy parsing of JSON logs. We have a few Data Engineering jobs that take hours just because of Python's single-threaded nature. I tried using multiprocessing, but the overhead of serializing data between processes almost canceled out the gains. If moving those specific modules to a Rust-based tool could cut that time down to minutes, it would save us a lot of money on our compute bill. I’m starting to look into PyO3 to see if I can just write the "hot" parts of the code in Rust while keeping the rest in Python.