I have built a decision tree for a customer churn project, but it is performing perfectly on training data while failing miserably on my test set. I suspect overfitting because the tree is becoming extremely deep. What are the specific hyperparameters I should tune in Scikit-Learn to stop the tree from growing too large, and is it better to prune the tree after it is fully grown or set constraints before training starts?
3 answers
Overfitting is the most common hurdle with decision trees because they naturally try to classify every single data point perfectly. To fix this, you should look into "Pre-Pruning" by adjusting hyperparameters like max_depth, min_samples_split, and min_samples_leaf. For instance, setting a max_depth of 5 or 10 usually prevents the model from capturing noise. Another powerful method is "Post-Pruning" using Cost Complexity Pruning ($ccp\_alpha$). In my experience working on financial risk models in 2023, finding the right alpha value through cross-validation helped me simplify my trees significantly while actually increasing the accuracy on the validation set.
Emily, that is a great breakdown of the pruning methods! Do you find that setting a min_impurity_decrease is more effective than just limiting the max_depth when dealing with very noisy datasets?
You might also want to consider using a Random Forest. Instead of relying on one deep tree, it averages multiple trees, which naturally reduces variance and handles the overfitting problem for you.
Jason is spot on. I used Random Forests in my last project and the stability it offers compared to a single decision tree is night and day, especially with high-dimensional data.
Mark, min_impurity_decrease is actually a more "intelligent" way to prune. While max_depth is a blunt force tool that stops the tree at a certain level regardless of the data, min_impurity_decrease ensures that a split only happens if it significantly improves the purity of the nodes. In a project I handled in late 2024, using a small threshold like 0.01 for impurity decrease helped me filter out insignificant features that were originally causing the model to hallucinate patterns where none existed.