I recently ran a project using automated machine learning and got a suspicious 99% accuracy. I’m pretty sure there is data leakage happening somewhere in the automated preprocessing. How do you identify which features are leaking info when the tool handles all the transformations behind the scenes?
3 answers
Data leakage in automated machine learning usually happens because the tool calculates statistics (like the mean for imputation) using the entire dataset before the train-test split. To diagnose this, look for features with an unusually high correlation to the target variable. A good trick is to run the model on a completely "out-of-time" validation set that the AutoML tool never touched. If the accuracy drops from 99% to 70%, you’ve found your leak. Always ensure your time-series data is split chronologically rather than randomly, as random splits in AutoML are a classic source of leakage.
Are you seeing high importance for a feature that would not actually be available at the moment of prediction in your automated machine learning model?
I always suggest manual exploratory data analysis before feeding everything into automated machine learning; it’s the only way to catch these issues early.
Beverly's advice is gold. You can't skip the "Data" part of Data Science just because the "Machine Learning" part is now automated.
Exactly! It turns out a "customer ID" was inadvertently acting as a proxy for the target because the data was sorted by the outcome before I uploaded it.