Title: Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models

Abstract: While large language models (LLMs) exhibit exceptional capabilities across a wide array of tasks, they frequently produce outputs that sound convincing yet contain factual errors. This issue is exacerbated by the absence of direct uncertainty estimates, leaving users unable to accurately gauge the trustworthiness of the model’s responses. Current techniques for quantifying uncertainty generally depend on indirect indicators, such as the entropy calculated from various sampled generations. These metrics are often hard to interpret and fail to fully capitalize on the model’s inherent capacity for self-evaluation. To address this, we introduce a straightforward and potent self-assessment technique for LLM uncertainty quantification. The proposed method organizes sampled generations into distinct semantic clusters, transforms these clusters into options for a structured multiple-choice question, and utilizes the probability the LLM assigns to each option as a measure of confidence. Empirical evaluations across diverse models and datasets reveal that our approach consistently surpasses baseline methods. Remarkably, it delivers competitive results with as few as two extra samples, highlighting both its efficacy and computational efficiency.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC