Title: The Mechanisms Behind How Large Language Models Handle Negation

Abstract: This research investigates the mechanistic processes Large Language Models (LLMs) employ to handle negation. We first demonstrate that while open-weight models frequently yield incorrect responses to queries involving negation, they actually contain internal components capable of processing it accurately. The low accuracy rates stem from attention behaviors in later layers that favor simplistic shortcuts; removing these specific attention modules significantly boosts performance on negation-based tasks. Second, we reveal the specific methods models use to process negation. We examined two primary hypotheses: either models utilize attention heads to target the negated phrase while suppressing associated concepts, or they directly build a representation for the entire negative phrase (for instance, creating a vector for "not gas" that encourages associations with liquids and solids). By applying various observational and causal interpretability techniques to Mistral-7B and Llama-3.1-8B, we found that LLMs utilize both mechanisms, though the "constructive" approach is more dominant. Together, these findings enhance our comprehension of LLM internals, emphasizing construction-heavy computations and the simultaneous presence of competing mechanisms.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC