I am building a predictive model for fraud detection where the positive class is very rare. My accuracy is 99%, but it's failing to catch actual fraud. Between Precision-Recall curves, F1-Score, and ROC-AUC, which should I prioritize to ensure my model is actually performing well on the minority class? I need a metric that reflects real-world business impact rather than just statistical probability.
3 answers
In fraud detection, "Accuracy" is a trap because a model can be 99% accurate by simply guessing "No Fraud" every time. You should prioritize the Precision-Recall (PR) Curve over ROC-AUC. ROC-AUC can be overly optimistic when dealing with heavily imbalanced classes. The F1-Score is a great middle ground as it provides the harmonic mean of precision and recall. However, if the cost of missing a fraud case (False Negative) is much higher than a false alarm, you should specifically optimize for Recall while maintaining a tolerable level of Precision for your team.
Have you considered using a Cost-Benefit matrix alongside your metrics? It allows you to assign a dollar value to False Positives versus False Negatives. What is the actual cost of a missed fraud?
I always use the AUPRC (Area Under the Precision-Recall Curve). It’s far more sensitive to the performance of the positive class than the standard ROC curve.
Exactly, Barbara. AUPRC is the gold standard for rare-event prediction. I used it for a churn project last year and it changed our entire perspective on model success.
That is a great point, James. For our bank, a missed fraud costs about $500 on average, while a false flag costs $10 in manual review time. By weighting the loss function in your XGBoost or Random Forest model to reflect this 50:1 ratio, the model will naturally prioritize the minority class. This shifts the focus from pure probability to actual business utility and ROI.