We recently deployed a lead scoring model that performed great in testing, but its accuracy has started to decay after just two months. I suspect "Data Drift" is the culprit. How do you guys monitor for drift in a production environment, and what are the triggers you use to decide when it's time to retrain the model? Are there specific Python libraries or tools you'd recommend for this?
3 answers
Model decay is a major part of the MLOps lifecycle. You should monitor two things: Data Drift (feature distribution changes) and Concept Drift (the relationship between features and target changes). I recommend using the 'Evidently' or 'Alibi Detect' libraries in Python. They allow you to run statistical tests like Kolmogorov-Smirnov to see if your incoming data distribution matches your training data. A good trigger for retraining is when your primary metric (like F1 or MAE) drops below a specific threshold for three consecutive days or when the drift score exceeds a set limit.
Do you have a feedback loop to get the "ground truth" labels quickly? If you don't get the actual outcomes for months, how are you even measuring the current accuracy?
I usually set up an automated pipeline that retrains the model every month anyway, just to capture the most recent seasonal trends. It's safer than waiting for a failure.
Scheduled retraining is a solid strategy, Karen. We do it quarterly and it has significantly reduced the variance in our monthly performance reports.
That's the struggle; our sales cycle is 90 days. In this case, you must rely on "Proxy Metrics." Monitor the distribution of the model's predictions (output drift). If the model suddenly starts scoring everyone 20% lower than last month, but the input data looks the same, your model is likely becoming obsolete. Tracking the stability of the input features is your first line of defense here.