I am currently working on a predictive modeling project with a dataset containing over a million rows, but nearly 30% of the entries have missing values across various features. Should I stick to simple imputation methods like mean/median, or is it better to use advanced techniques like MICE or KNN? I am worried about introducing bias into the model during the preprocessing stage.
3 answers
When dealing with a dataset of this size, the first step is to analyze the "mechanism" of missingness—whether it is Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR). For MAR data, Multiple Imputation by Chained Equations (MICE) is generally superior to simple mean or median imputation because it preserves the statistical relationships between variables. Simple imputation often artificially reduces the variance of your data, which can lead to overconfident and biased models. If you have the computational resources, I’d suggest a random forest-based imputer like MissForest, as it handles non-linear relationships and categorical variables exceptionally well without requiring heavy feature scaling.
That is a very thorough explanation, but have you considered the performance cost of using MICE on a million-row dataset? Wouldn't a simpler iterative imputer be more efficient for production environments where inference speed is a priority, or do you think the accuracy gain always outweighs the latency?
For very large datasets, sometimes the most honest approach is to treat "Missing" as its own category or use models like XGBoost that have built-in handles for null values.
I agree with Daniel. Using an algorithm that naturally understands missingness saves a lot of time in the feature engineering pipeline and often results in higher validation scores.
Steven, you raise a valid point regarding production latency. In my experience, you can perform the heavy imputation during the training phase to build a robust model, but for live inference, you might need a simplified strategy. However, if the missing data is in critical features, sacrificing a few milliseconds for a MICE-based prediction is usually the safer bet to ensure the integrity of the business insight.