I’ve mostly seen Decision Trees used for binary classification, like spam detection. Can I use them to predict a continuous value, like the price of a house or the temperature? How does the "purity" metric change when the target is a number instead of a class label? I’m trying to decide if a Regression Tree is a better alternative to a simple Linear Regression for a non-linear dataset I'm working on.
3 answers
Yes, they are great for regression! Instead of Gini or Entropy, Regression Trees typically use Mean Squared Error (MSE) or Mean Absolute Error (MAE) as the splitting criterion. The algorithm looks for the split that minimizes the variance of the target values within each resulting node. The prediction at a leaf node is usually the average value of all samples in that leaf. In a project I did in late 2023 involving real estate pricing, the Decision Tree outperformed Linear Regression because it could capture non-linear relationships and interactions between features without me having to manually create interaction terms.
Margaret, that’s helpful! One concern I have is that Regression Trees predict a constant value for each leaf. Doesn't that make the output look like a "staircase" rather than a smooth curve?
Just keep in mind that trees cannot extrapolate. If you have house prices higher than anything in your training set, the tree will just predict the highest value it has ever seen.
That extrapolation point is critical! I learned that the hard way when my demand forecasting model failed during a record-breaking sales month. Linear models are better if you expect values outside the training range.
Thomas, you are absolutely right. Regression trees are "piecewise constant." If you plot the predictions, it looks like a series of steps. This can be a downside if you need a very smooth prediction. However, in my 2024 experiments, I found that you can smooth this out by using an ensemble of trees (like Gradient Boosting) or by increasing the number of leaves. For most real-world data where trends aren't perfectly smooth anyway, the "staircase" effect is rarely a dealbreaker for accuracy.