Title: Treating Hallucinations as Orthogonal Noise: Aligning Inference-Time Manifolds Through Dynamic Contextual Orthogonalization

Abstract:

The generation of information that conflicts with contextual facts or logical constraints—a phenomenon known as hallucination in Large Language Models (LLMs)—continues to hinder their reliable real-world deployment. To tackle this persistent issue, our study introduces a geometric perspective grounded in the linear representation hypothesis. We posit that hallucinations should be understood as noise vectors that are orthogonal to the semantic manifold of the residual stream. In this view, while attention heads are designed to propagate information that aligns with the context subspace, hallucinations occur when certain heads inject components that are perpendicular to this subspace, thereby destabilizing the latent representation’s coherence.

Building on this theoretical foundation, we present Dynamic Contextual Orthogonalization (DCO), a novel intervention technique applied during inference. DCO leverages the input residual stream as a dynamic anchor for context, enabling the orthogonal decomposition of attention head outputs. To separate meaningful semantic updates that align with the context from divergent noise, the method implements a layer-wise Z-score suppression mechanism. This mechanism statistically identifies and attenuates outlier orthogonal components.

We evaluated DCO on Llama-3 models (both 8B and 70B) using benchmarks including XSum, NQ-Swap, and IFEval. The results indicate that DCO delivers superior contextual faithfulness when compared to current state-of-the-art intervention baselines. Additionally, DCO preserves high performance on knowledge-dependent tasks such as TriviaQA and TruthfulQA. This success is significant because it effectively resolves the common trade-off seen in existing methods, where suppressing hallucinations often leads to a loss of parametric knowledge. Our findings confirm the geometric interpretation of hallucinations and position DCO as a computationally efficient strategy for enforcing manifold alignment. The source code for this project is accessible at https://github.com/Harry-Miral/DCO.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC