Our team is working with high-dimensional datasets for credit scoring, and we are debating between using XGBoost and LightGBM. Given the noise in financial data, which ensemble methods or regularization techniques would you recommend to prevent overfitting while maintaining high interpretability?
3 answers
For financial datasets, LightGBM is often superior due to its speed and ability to handle large-scale data, but XGBoost’s recent updates have made it very competitive in terms of accuracy. To handle high dimensionality, I suggest using Lasso (L1) regularization within your gradient boosting framework. Additionally, consider using SHAP values for interpretability, as stakeholders in finance need to understand why a credit score was assigned. Don’t ignore the power of CatBoost if you have categorical variables, as it handles them natively without the need for extensive one-hot encoding.
Have you considered performing a Principal Component Analysis (PCA) before feeding the data into your ensemble models to reduce the feature space first?
I’ve found that a Stacking Classifier, combining a Random Forest with a Gradient Boosting Machine, often provides a more robust result than any single model alone.
Stacking is excellent! I’ve used a logistic regression as the meta-learner in a stack to keep the final output somewhat interpretable and stable.
PCA is useful, but we are worried about losing the "meaning" of individual features which is required for regulatory compliance. We need to keep the original features intact as much as possible.