Title: Beyond Mere Inquiry: The Crucial Role of Protocol Sensitivity in LLM Confidence Calibration

Abstract:

Evaluating large language model (LLM) confidence calibration typically involves juxtaposing token-probability scores with verbalized confidence estimates. While these two signals are frequently interpreted as direct indicators of model uncertainty, such comparisons rely on specific measurement decisions that are often left implicit. This study fixes the method for eliciting verbalized confidence—utilizing a single prompt template, probability scale, and output format—while systematically varying the parameters that govern the verbalized-versus-token comparison. Specifically, we examine which answer string is assigned the token-probability score, the method used to extract this score from the answer tokens, and the conditioning context under which the measurement occurs.

Our evaluation spans four question-answering benchmarks and three distinct families of open-source 7–8B base and Instruct models, supplemented by robustness checks using larger Qwen2.5 variants within the same family. The analysis reveals that these comparisons are highly sensitive to the chosen protocols. Altering the conditioning context can shift the sign or magnitude of the Expected Calibration Error (ECE) gap across different settings, while variations in token readout methods induce smaller but still directionally significant changes. Conversely, modifying the ECE estimator yields negligible effects. Under a default protocol involving generated answers and bare context, Instruct models demonstrate near-parity with base models, rather than exhibiting the substantial calibration improvements often attributed to verbalized confidence.

Furthermore, our separate analysis of supplied answers indicates that plausible but incorrect responses receive confidence levels nearly identical to those of provided gold-standard answers. This suggests that verbalized confidence captures not only correctness but also answer plausibility and provenance. We contend that both confidence metrics should be regarded as protocol-dependent behavioral measurements. To address this, we propose a reporting checklist that details elicitation provenance, the scored answer, token-probability readout methods, and conditioning context.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC