With Python 3.12 out and the ongoing discussions about the Global Interpreter Lock (GIL) being optional in future versions, how is your dev team preparing? We rely heavily on multi-threading for data processing, and I'm wondering if we should stick to multiprocessing or wait for the "nogil" builds to mature.
3 answers
We are currently sticking with the multiprocessing module for any heavy CPU-bound tasks. While the "nogil" movement is exciting, it's still experimental and could introduce subtle thread-safety bugs in third-party C-extensions like NumPy or Pandas. For Python 3.12, we’ve focused more on leveraging the new f-string improvements and the performance boosts in the garbage collector. My strategy is to wait until at least 3.13 or 3.14 before we even consider refactoring our core threading logic to rely on a GIL-free environment.
Have you noticed any significant performance regressions in 3.12 with older libraries, or has the backward compatibility been as solid as advertised?
We actually moved our most performance-critical paths to Rust via PyO3. It’s much safer than waiting for the GIL changes to solve our concurrency issues.
I agree with Deborah. Using Rust for the heavy lifting while keeping the business logic in Python is the "Gold Standard" right now. It completely bypasses the GIL headache.
I agree with Deborah. Using Rust for the heavy lifting while keeping the business logic in Python is the "Gold Standard" right now. It completely bypasses the GIL headache.