I'm struggling with a model that performs perfectly on my training data but has a 20% error rate on the test set. I know this is overfitting, but I don't quite understand the relationship between model complexity and the bias-variance tradeoff. How can I find the "sweet spot" where my model generalizes well?
3 answers
Think of bias as the error from erroneous assumptions in the learning algorithm (underfitting), and variance as the error from sensitivity to small fluctuations in the training set (overfitting). High variance means your model is "memorizing" noise rather than learning patterns. To find the sweet spot, you should use Cross-Validation to monitor both training and validation errors. As you increase model complexity, the training error always goes down, but the validation error will eventually start to rise. That "U-shape" inflection point on the validation curve is exactly where you want to stop training or apply regularization.
Have you tried applying L1 or L2 regularization (Lasso/Ridge) to penalize the model for having excessively large weights?
Learning curves are your best friend here. Plotting error vs. training set size will tell you if you need more data or a simpler model architecture.
Mitchell is spot on. If the training and validation curves have a large gap, you definitely have a high-variance problem that needs more regularization.
I’ve tried L2 regularization, Byron, but I’m unsure how to pick the right "lambda" or alpha value. Is there a systematic way to tune the regularization strength without just guessing and checking? I'm worried that if I set it too high, I'll end up with a high-bias model that can't even learn the basic trends in the data.