My decision tree model has a feature_importances_ attribute, but I’m not sure how to read the values. Some features have high scores while others are zero. Does a score of zero mean the feature is completely useless, or just that it wasn't chosen for a split? Also, if two features are highly correlated, how does the tree decide which one gets the "importance" score? I want to use this for feature selection in a larger pipeline.
3 answers
Feature importance in trees (often called Gini Importance) is calculated by looking at how much each feature decreases the total impurity of the model. If a feature has an importance of zero, it means it was never used to make a split, which usually implies it provided less information than other available features. In 2023, I was working on a medical dataset and noticed that if two features are highly correlated, the tree might pick one and ignore the other entirely. This doesn't mean the second feature is "useless," just that its information was already captured by the first one. Always check for collinearity before relying solely on these scores!
Nancy, if the importance scores are biased toward features with many unique values (high cardinality), is there a better way to measure importance that is more reliable? POSTED BY : Daniel Harris
I find that visualizing the tree itself using plot_tree is the best way to see importance. If a feature is the root node, it’s clearly the most significant one!
Great tip, Brenda. Seeing the top 3 levels of the tree often tells a better story than a bar chart of importance scores, especially when explaining results to non-technical stakeholders.
Daniel, you should definitely look into "Permutation Importance." It works by shuffling a single feature and seeing how much the model accuracy drops. It is much more robust against the high-cardinality bias that native Gini importance has. When I audited a marketing model in early 2024, Permutation Importance revealed that a "Customer ID" column was getting high Gini importance simply because it was unique, even though it had zero predictive power. Switching methods saved us from a huge mistake!