Title: Large Language Models Fail to Maintain Contextual Invariance

Abstract: Conventional evaluation methodologies operate on the premise that the responses generated by large language models (LLMs) remain consistent when prompts are situated within discourses that are contextually equivalent. This study challenges that assumption by examining gender inference tasks. Through a controlled experiment involving pronoun selection, we incorporated minimal discourse contexts that offer no theoretical insight, yet discovered that these additions trigger significant and systematic alterations in model outputs. While correlations with cultural gender stereotypes—observable in decontextualized environments—diminish or vanish when context is applied, features theoretically irrelevant to the task, such as the gender of a pronoun referring to an unrelated entity, emerge as the strongest predictors of model behavior. A Contextuality-by-Default analysis indicates that this dependency remains in 19% to 52% of instances across various models, even after controlling for all marginal contextual effects on individual outputs, thereby ruling out simple pronoun repetition as the cause. These results demonstrate that LLM outputs breach contextual invariance even when syntactic structures are nearly identical, raising important concerns for bias assessment and the deployment of these systems in high-stakes environments.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC