Abstract
Explainable AI (XAI) is essential for trust and transparency in deep learning, especially in medical imaging.
Existing local explanation methods provide per-instance insights but fail to show whether similar explanations
hold across samples of the same class. This limits global interpretability and demands time-consuming manual
review by clinicians to trust models in practice. We introduce the Consensus Alignment Score (CAS), a
novel metric that quantifies consistency of explanations at the class level. We also present ConsensusXAI, an
open-source, model- and method-agnostic framework that evaluates explanation agreement quantitatively (via
CAS) and qualitatively (through consensus heatmaps) per class. Unlike prior benchmarks, ConsensusXAI uses
a latent-space clustering approach, Latent Consensus, to identify dominant explanation patterns, exposing
biases and inconsistencies towards certain classes. Evaluated across two different medical imaging modalities for
both correct and incorrect predictions on two different backbones, our method consistently reveals meaningful
class-level insights, outperforming traditional consensus method i.e. SSIM, and enabling faster, more confident
clinical adoption of AI models.
Existing local explanation methods provide per-instance insights but fail to show whether similar explanations
hold across samples of the same class. This limits global interpretability and demands time-consuming manual
review by clinicians to trust models in practice. We introduce the Consensus Alignment Score (CAS), a
novel metric that quantifies consistency of explanations at the class level. We also present ConsensusXAI, an
open-source, model- and method-agnostic framework that evaluates explanation agreement quantitatively (via
CAS) and qualitatively (through consensus heatmaps) per class. Unlike prior benchmarks, ConsensusXAI uses
a latent-space clustering approach, Latent Consensus, to identify dominant explanation patterns, exposing
biases and inconsistencies towards certain classes. Evaluated across two different medical imaging modalities for
both correct and incorrect predictions on two different backbones, our method consistently reveals meaningful
class-level insights, outperforming traditional consensus method i.e. SSIM, and enabling faster, more confident
clinical adoption of AI models.
| Original language | English |
|---|---|
| Number of pages | 12 |
| Publication status | Accepted/In press - 11 Nov 2025 |
| Event | IEEE/CVF Winter Conference on Applications of Computer Vision 2026 - JW Marriott Starpass, Tucson, United States Duration: 6 Mar 2026 → 10 Mar 2026 https://wacv.thecvf.com/ |
Conference
| Conference | IEEE/CVF Winter Conference on Applications of Computer Vision 2026 |
|---|---|
| Country/Territory | United States |
| City | Tucson |
| Period | 6/03/26 → 10/03/26 |
| Internet address |