Title: Reducing Hallucinations in Large Language Models by Bypassing Specific Decoder Layers

Abstract:

While Large Language Models (LLMs) have demonstrated impressive capabilities across a wide array of natural language processing tasks, they remain susceptible to generating hallucinated content—outputs that contradict established facts. This study presents a thorough layer-by-layer examination of the decoding mechanism, uncovering that hallucinations frequently stem from the deeper layers of the network. To counteract this phenomenon, we propose DeLask (Decoder Layer Skipping), a new decoding approach that selectively bypasses layers identified as high-risk for hallucination generation.

DeLask is grounded in the theoretical perspective that the forward pass of an $L$-layer Transformer can be viewed as conditionally equivalent to $L$ iterations of gradient descent. Within this framework, we introduce a metric called the \emph{driftance value}, which is calculated by measuring the cosine similarity between gradients produced in successive decoder steps. This metric allows the system to pinpoint layers where the optimization direction effectively reverses, signaling potential errors. Instead of completely removing these problematic layers, DeLask employs a strategy of partial aggregation, blending their hidden states with those of the preceding layers. This technique helps maintain structural consistency while dampening incorrect signals. Our extensive testing across various LLM architectures and benchmark datasets confirms that DeLask reliably reduces hallucinations and boosts overall model trustworthiness, offering a lightweight and broadly applicable solution for enhancing the resilience of large-scale language models.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC