Title: Caliper: Investigating Lexical Anchors Against Causal Structure in LLMs

Abstract

While large language models achieve accuracy rates between 50% and 70% on causal reasoning datasets like CLadder, it remains ambiguous whether these results stem from genuine structural reasoning or simple lexical pattern matching. To address this, we present Caliper, a method of controlled perturbation that swaps semantic variable names for placeholder tokens. This approach maintains the underlying causal graph and probabilistic specifications of each query while obscuring lexical cues.

Our evaluation across three causal reasoning benchmarks and nine instruction-tuned LLMs (ranging from 3.8B to 671B parameters) reveals significant performance declines due to lexical anonymization. On a local subset of models sized 3.8B to 14B, accuracy drops were recorded at +7.6, +27.0, and +11.1 percentage points. These effects intensify when assessing nine frontier models from the 2024-2026 generation periods, where accuracy gaps reached +29.6 and +18.0 percentage points on the CRASS and e-CARE benchmarks, respectively.

Analysis of 40 model-by-benchmark combinations shows that 39 exhibit a positive performance gap. Notably, this discrepancy shrinks by a factor of 17x on the pseudoword subset of CLadder. Although structured scaffolding and few-shot in-context learning help reduce this gap, the improvement primarily stems from decreased P0 accuracy in smaller models rather than a recovery of P1 performance. Ultimately, when evaluated in a zero-shot setting, current instruction-tuned LLMs demonstrate minimal evidence of structural causal reasoning once lexical anchors are eliminated.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC