Title: DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations

Abstract:

Current approaches to categorizing Large Language Model (LLM) hallucinations focus on the nature of the error, such as memorized misconceptions, reasoning breakdowns, or fluent fabrications. While these classifications aid in diagnosis, they fail to address a critical operational question: which uncertainty scoring mechanism is capable of identifying a specific error? To bridge this gap, we introduce DECK, a complementary taxonomy that classifies errors based on their detectability signature—essentially, the signal a particular scorer family would perceive.

DECK organizes errors into a 2x2 matrix defined by inter-sample consistency and token-level confidence, resulting in four distinct behavioral regimes: Drift, Entrenched, Confabulation, and Knotted. Each regime corresponds to specific detection capabilities: black-box consistency scorers are effective for Drift (D) and Confabulation (C); white-box token-probability scorers target Knotted (K) and Confabulation (C); and only an LLM-as-a-Judge with independent pretraining can identify Entrenched (E) errors. We determine cell membership by applying a Youden’s J optimal split on each scorer axis.

We validated this taxonomy across three models and four datasets using two methods: analyzing scorer-pair disagreement and verifying that external labels (such as SelfAware unanswerable, HaluEval adversarial, and PopQA entity popularity) align with their predicted DECK cells, including secondary-cell refinements based on model scale and content specificity. Additionally, we identified a universal blind spot in output-level uncertainty quantification (UQ). On inputs involving knowledge gaps, where generators produce confident and repeatable fabrications, all output-level families fail by design. Furthermore, a linear probe on Llama-3-8B’s hidden states performed at chance levels, suggesting this failure mode may persist at the activation level. Consequently, richer internal-state methods, such as UQ heads and information-theoretic estimators, remain to be tested.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC