Auditing Our Models for Visual Explainability
Is learned attention sufficient for faithful and well-localized heatmaps?
A heatmap can reflect what a model computed and still point at the wrong anatomy. We score our models' explanations on localization and faithfulness, and find that learned attention holds up on both.
Abstract
Saliency heatmaps derived from internal mechanisms are routinely used to justify the predictions of radiology ML models, but a heatmap that reflects the computation behind a prediction need not be anatomically reasonable. In this post, we evaluate the interpretability of our CR pathology classification and radiology report generation models. Specifically, we evaluate the various heatmap explanations produced by our models along two axes (localization and faithfulness) and find that learned attention weights provide useful visual explainability information that is strongly rooted in the model's internal reasoning.
Cite
@misc{mishra2026auditing,
title = {Auditing Our Models for Visual Explainability},
author = {Aakash Mishra and Arjun Karpur},
year = {2026},
howpublished = {Epsilon Health Research},
url = {https://www.epsilon.health/research/auditing-model-attention}
}