I'm working on a fraud detection project where the minority class is only 0.1% of the dataset. My XGBoost model has 99.9% accuracy but a terrible recall score for actual fraud cases. Should I focus on oversampling with SMOTE, or are there specific loss functions in Data Science libraries that handle this better without creating synthetic bias?
3 answers
Accuracy is a trap in fraud detection; you should be looking at the Precision-Recall AUC or the F1-Score instead. While SMOTE is a classic approach, it often creates "noise" by generating synthetic points in overlapping feature spaces. A cleaner approach in XGBoost is using the scale_pos_weight parameter, which essentially tells the model to penalize mistakes on the minority class more heavily. For deep learning, you could implement a Weighted Cross-Entropy loss. This allows the model to learn the underlying patterns of the rare class without the risk of overfitting to the artificial data points created by oversampling.
I’ve had similar issues with SMOTE; have you tried ADASYN or Tomek Links to clean up the overlapping boundaries? Also, are you using an isolation forest as a baseline? Sometimes treating fraud as an anomaly detection problem rather than a classification problem yields much better results when the imbalance is that extreme.
Focus on your feature engineering. Usually, the "time since last transaction" or "distance from home" features are more important than the specific algorithm you use for imbalance.
Jennifer is right. I’ve seen that adding "velocity" features—like the number of transactions in the last hour—often makes the class imbalance much easier for the model to separate, regardless of whether you use SMOTE or not.
Michael, I tried an Isolation Forest, but it struggled with the high dimensionality of our feature set. Regarding your question on ADASYN, it did help slightly with recall, but it pushed the false positive rate too high for our business team to accept. We actually found that using a combination of Tomek Links to remove the majority class noise and then applying the cost-sensitive learning Emily mentioned gave us the most stable ROC-AUC curve so far.