We are using a complex Neural Network for medical diagnosis, but the doctors won't trust it unless we can explain why a certain prediction was made. I’ve looked at SHAP and LIME. Which one is better for image-based data vs. tabular data in a clinical setting? I need something that can highlight specific features or pixels that influenced the output.
3 answers
For tabular data, SHAP (SHapley Additive exPlanations) is generally preferred because it’s based on solid game theory and provides consistency that LIME lacks. It tells you exactly how much each feature contributed to the deviation from the average prediction. For image-based Deep Learning, you should look into Gradient-weighted Class Activation Mapping (Grad-CAM) or Integrated Gradients. These create "heatmaps" over the original image, showing the doctor which part of an X-ray, for instance, triggered the "pneumonia" classification. SHAP also has a "DeepExplainer" specifically for Keras/PyTorch that works well, but it can be computationally expensive compared to LIME’s local approximations.
How are you handling the "Stability" of these explanations? I’ve found that LIME can sometimes give completely different explanations for two very similar patients, which might destroy the doctors' trust rather than build it. Have you looked at any "Global" explanation methods?
Just a warning: these tools explain the model, not the truth. If your model is biased, SHAP will just explain the bias, not provide a correct medical diagnosis.
Exactly, Megan. We use SHAP as a debugging tool first. If the heatmap shows the model is looking at a watermark on the X-ray instead of the lungs, we know we have a data leak.
Thomas, that instability is exactly why we moved away from LIME. We are currently testing "KernelSHAP" which is slower but much more consistent across similar inputs. James, for your medical use case, I’d suggest presenting both a global view (what features the model values in general) and a local view (the specific patient). Doctors seem to respond better to "Feature Importance" charts that align with their own clinical knowledge, like seeing "Blood Pressure" as a top driver for heart disease risk.