I am working on a competition dataset with over 5 million rows and 100 features. I’ve traditionally used XGBoost, but I am hearing that LightGBM is significantly faster for large-scale data. What are the primary trade-offs in terms of memory usage, training speed, and predictive accuracy between these two GBDT frameworks?
3 answers
LightGBM is almost always faster for datasets of that size because of its "Leaf-wise" growth strategy and Gradient-based One-Side Sampling (GOSS). While XGBoost grows trees level by level, LightGBM grows them leaf by leaf, which reduces loss much more quickly. More importantly, LightGBM uses "Exclusive Feature Bundling" to reduce the number of features, making it incredibly memory efficient. In my experience, XGBoost 2.0 has closed the speed gap significantly, but LightGBM still holds the edge when you are constrained by RAM or need very fast iteration cycles during hyperparameter tuning.
Since you have 5 million rows, are you using a GPU-accelerated version of these libraries, or are you strictly sticking to CPU-based training?
Don't forget that LightGBM is highly sensitive to overfitting on smaller datasets. With 5 million rows you are safe, but always tune the 'num_leaves' carefully.
Bridgette is right. In LightGBM, 'num_leaves' should be smaller than 2^(max_depth) to prevent the trees from growing too deep and complex.
I’m currently on a CPU-only server with 64GB of RAM. Franklin, do you think the histogram-based algorithm in LightGBM will still outperform XGBoost in this environment, or is the GPU support the main reason people make the switch? I've noticed XGBoost can be quite a memory hog when it builds the initial histograms for such a high row count.