I'm working on a fraud detection project, and my "fraud" class is only 0.5% of the total data. My machine learning model has high accuracy but terrible recall. What techniques are you guys using beyond simple oversampling to make sure the minority class is actually being identified?
3 answers
Standard accuracy is a trap for imbalanced datasets. You should immediately switch your evaluation metrics to Precision-Recall AUC or the F1-Score. Beyond SMOTE, I highly recommend looking into cost-sensitive learning. Most modern algorithms, like XGBoost or LightGBM, allow you to pass a "scale_pos_weight" parameter. This tells the model to penalize misclassifications of the minority class more heavily during training. Also, consider using Anomaly Detection techniques rather than standard classification. If the fraud is that rare, treating it as an outlier might yield much better results than trying to force a supervised fit.
Have you tried different data augmentation steps? Sometimes generating synthetic samples can lead to overfitting if not handled carefully. What kind of feature engineering have you done to separate the classes?
SMOTE is often overused. Try EasyEnsemble or BalancedRandomForest. They under-sample the majority class in a more intelligent way that preserves the underlying data distribution.
Lisa makes a great point. BalancedRandomForest is excellent because it’s much more robust to noise than simple oversampling techniques which can sometimes just amplify the errors.
Jeffrey, I’ve done some basic aggregations, but the features still overlap a lot. I’m currently looking into whether I can use autoencoders to learn the "normal" behavior and then flag the fraud as high-reconstruction error. It feels like a more natural way to handle the 0.5% class than just duplicating rows over and over. I'll definitely check out that weight parameter Rebecca mentioned too.