I’ve been an R user for years because data.table is incredibly fast for multi-gigabyte files. However, with the release of Pandas 2.0 and the Apache Arrow integration, I’m hearing that Python’s speed has improved significantly. For those of you handling big data on local machines, has Python finally closed the performance gap for data manipulation?
3 answers
While Pandas 2.0 with the Arrow backend is a massive leap forward, R’s data.table still holds the crown for memory efficiency and raw speed on large-scale in-memory operations. The syntax of data.table is much more concise for complex grouping and joining. However, the gap is definitely closing. If you are working in an environment where you need to pass data between different languages, Python’s integration with Arrow makes it much faster than it used to be. But if I have a 20GB CSV and need to do a grouped aggregation, I’m still reaching for R every single time.
Have you looked into Polars as a middle ground? It’s available for both languages and seems to be outperforming both Pandas and data.table in several recent benchmarks.
Speed is important, but readability matters too. I still find the Tidyverse/Dplyr syntax in R to be the most human-readable way to describe data transformations.
Readability is a major factor, Patricia. When working in teams, having code that others can easily audit and understand is often worth a slight hit in execution speed.
Michael, Polars is actually incredible. I’ve started using the Python version for my ETL pipelines and it’s significantly faster than Pandas for large joins. It’s written in Rust, so it handles multi-threading much better than the older Python libraries.