I am working on a project with a lot of "dirty" data where several columns have up to 30% missing values. Should I just drop these rows, or is mean/median imputation sufficient? I've heard about MICE and KNN imputation but I'm worried about the computational cost on a dataset with millions of records. What is the industry standard for this?
3 answers
Handling missing data is an art. If the data is Missing Completely at Random (MCAR), you might get away with simple imputation. However, if there's a pattern to the missingness, simple mean imputation will bias your model. For large datasets, dropping rows is usually a last resort because you lose valuable information. I suggest looking into "Indicator Variables" where you flag the missingness as a feature itself. For high-scale data, iterative imputer (similar to MICE) in Scikit-Learn is quite robust. Always validate your approach by checking if the distribution of the feature changes significantly after imputation.
Have you checked if the missing values are concentrated in specific segments of your target population or are they uniform?
For millions of rows, complex methods like KNN are too slow. Use simple imputation or XGBoost which handles missing values internally.
I agree with Kimberly. Modern gradient boosting algorithms are surprisingly good at learning the best direction for missing values during the training phase.
They seem to be concentrated in the older demographic. That's a huge red flag! It means your data is Missing Not at Random (MNAR). If you just impute the average, you'll misrepresent that entire age group. In this case, you should try to find why the data is missing or use more advanced modeling that accounts for this systematic bias. Domain knowledge is often more important here than the actual algorithm used.