Title: Domain-Shift-Aware Conformal Prediction for Large Language Models

Abstract:

Despite the remarkable success of large language models (LLMs) across a wide array of tasks, their propensity for generating factually inaccurate yet highly confident responses—commonly referred to as hallucinations—presents significant challenges for real-world deployment. While conformal prediction offers rigorous, distribution-free coverage guarantees with finite samples, its standard application often fails when subjected to domain shifts, resulting in unreliable prediction sets and insufficient coverage. To address this limitation, we introduce Domain-Shift-Aware Conformal Prediction (DS-CP), a novel framework designed to adapt conformal prediction mechanisms for LLMs operating under domain shift. This approach enhances adaptivity and preserves validity by systematically adjusting the weights of calibration samples according to their proximity to the test prompt. Both our theoretical analysis and empirical evaluations on the MMLU benchmark indicate that DS-CP achieves more reliable coverage compared to traditional conformal prediction methods, particularly in scenarios involving significant distributional shifts, all while maintaining computational efficiency. Consequently, this work represents a practical advancement toward enabling trustworthy uncertainty quantification for LLMs in practical applications.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC