Our fraud detection model’s accuracy has plummeted over the last three months. I suspect "Concept Drift" because consumer behavior is shifting rapidly. How do you monitor for this in production, and what's the best strategy for retraining without losing historical context?
3 answers
Concept Drift is a silent killer. You need to monitor the statistical distribution of your incoming features, not just the model's accuracy. We use the Kolmogorov-Smirnov (KS) test to compare the "live" data distribution against the "training" data. If the p-value drops, we trigger an alert. For retraining, don't just dump the old data. Use a "Weighted Retraining" approach where you give more importance to the most recent 30 days of data. This allows the model to adapt to the new "concept" of fraud while still remembering the fundamental patterns it learned previously.
Do you think it’s "Data Drift" (changes in input) or "Label Drift" (the relationship between input and target has changed)? The solution is different for each.
Use a "Sliding Window" for your training set. As new data comes in, the oldest data drops out. This keeps your model's "memory" fresh and relevant.
Great point, Nancy. The sliding window is simple but effective. It’s the most straightforward way to ensure the model doesn't become a dinosaur in a fast-moving market.
William, it looks like both. The types of transactions are changing, and so is the way fraudsters bypass our checks. We’ve started using an "Ensemble" approach. We keep the base model trained on historical data but add a "Challenger" model that is trained only on the last two weeks. We compare their performance in real-time. If the Challenger consistently wins, we promote it to the "Champion" position. This automated pipeline keeps our fraud detection agile without constant manual intervention.