I'm training a CNN-LSTM model to detect credit card fraud, but only 0.1% of my data consists of actual fraud cases. The model keeps achieving 99.9% accuracy by just predicting 'no fraud' for everything. I've tried oversampling, but it's overfitting. What are the best loss functions or architectures to fix this?
3 answers
Have you considered using an Anomaly Detection approach like Autoencoders instead of a standard supervised classifier for this specific problem?
Try cost-sensitive learning. You can manually assign a higher "penalty weight" to the fraud class in your model's configuration so the optimizer treats a missed fraud as a much bigger error.
I agree with Diane. Sometimes the simplest solution is just a weighted loss function. It’s much easier to implement in Keras or PyTorch than building complex synthetic data pipelines.
Accuracy is a "vanity metric" for imbalanced data; you need to focus on the F1-Score or the Area Under the Precision-Recall Curve (AUPRC). Instead of simple oversampling, I highly recommend using Focal Loss. It adds a factor to the standard Cross-Entropy loss that down-weights the loss for well-classified (easy) examples, forcing the model to focus on the rare, difficult fraud cases. You could also try Synthetic Minority Over-sampling Technique (SMOTE), but apply it only to the training set and keep your validation set "natural" to ensure the model generalizes well to real-world ratios.
Brandon, that’s a sharp suggestion. An Autoencoder trained only on "normal" transactions will naturally have a high reconstruction error when it encounters fraud. To answer your point about architecture: I’ve found that combining an Autoencoder for feature extraction with a light Gradient Boosting Machine (GBM) on the errors works better than a pure LSTM when the sequence length varies. It captures the "weirdness" of the transaction without needing a massive amount of labeled fraud examples.