Title: Leveraging Ambiguity to Enhance Error Prediction Through Uncertainty Quantification

Abstract: Predicting whether a model’s output is accurate—known as Error Prediction—is frequently addressed using Uncertainty Quantification (UQ). However, while UQ metrics effectively identify instances where a model lacks the knowledge or capacity to generate a response, they also conflate this with aleatoric uncertainty, which stems from inherent ambiguity in the input data and context. This study introduces a technique to refine error prediction for Large Language Models (LLMs) by separating input ambiguity from the UQ signal. Through experiments involving Question Answering (QA) tasks and six distinct UQ metrics, we demonstrate that these metrics are more effective at predicting errors in unambiguous instances compared to questions with multiple valid answers. To address this, we integrate gold-standard and predicted ambiguity labels into the error prediction workflow using Gated Experts and Selective Prediction. Our results indicate that incorporating ambiguity data enhances error prediction performance across various model architectures, training and evaluation methods, datasets (even those presumed to be unambiguous), and sources of aleatoric uncertainty. Specifically, this approach boosts the Partial Receiver Operating Characteristic (PRR) score by more than 10 points for individual UQ metrics on standard datasets.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC