Title: Enhancing RAG Interpretability: Contrastive Evidence Retrieval via Interpretable Attention Alignment

Abstract:

Addressing the critical challenges of factuality and interpretability in Retrieval-Augmented Generation (RAG) systems remains an urgent, unresolved issue. To this end, we propose Contrastive Evidence Rationale Attention (CERA), a novel retrieval framework that distinguishes itself by integrating subjectivity-driven hard negative selection with an evidential inductive bias within contrastive learning, facilitated by an auxiliary attention alignment loss. CERA optimizes a dense retriever through a dual-objective training process: triplet-based contrastive learning and interpretable attention alignment. The latter guides CLS-to-token attention by applying a part-of-speech-weighted masking distribution derived from human-annotated factual rationales as evidence signals.

Evaluated on a substantial dataset of clinical trial reports, our results indicate that subjectivity-based hard negative selection significantly boosts retrieval efficacy outperforming both Contriever and standard hard negative selection baselines. Additionally, the rationale alignment mechanism enhances faithfulness without compromising competitive retrieval performance. These findings substantiate the hypothesis that attention mechanisms provide more reliable explanations of model behavior when directed by human rationales. By transcending mere topical similarity, CERA empowers retrievers to pinpoint specific tokens that form supporting evidence, thereby fostering greater interpretability in evidence selection for RAG architectures.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC