I'm working on a credit card fraud project where only 0.1% of transactions are fraudulent. My model is getting 99.9% accuracy just by predicting "Not Fraud" every time! How do I force the model to prioritize the minority class without creating too many "False Positives" for our customers?
3 answers
Stop looking at "Accuracy" immediately; it's a useless metric for imbalanced data. You should focus on the "Area Under the Precision-Recall Curve" (AUPRC). To improve the model, try "Cost-Sensitive Learning" by assigning a much higher penalty to misclassifying a fraud case than a legitimate one. Many XGBoost and LightGBM implementations have a scale_pos_weight parameter for this. While oversampling techniques like SMOTE are popular, they can lead to overfitting. I prefer using "Balanced Random Forests" or simply undersampling the majority class to create a more even training environment.
If we use undersampling, aren't we throwing away valuable data that could help the model understand the nuances of legitimate transactions?
You should try "Anomaly Detection" algorithms like Isolation Forest or One-Class SVM. They treat fraud as an outlier rather than a classification category.
That's a great suggestion, Linda. Isolation Forests are particularly effective because they don't require the data to be balanced to find the "weird" transactions.
James, that's a valid worry. To answer that, you should look into "EasyEnsemble" or "BalancedBagging." Instead of throwing data away, these methods create multiple subsets of the majority class and pair each with the minority class. You train a separate model on each subset and then ensemble them. This way, the model eventually "sees" all the legitimate data across the different iterations, but each individual learner isn't overwhelmed by the 99.9% majority. It gives you the best of both worlds.