Title: Selective Reading: A Curvature-Conditioned Query for Linear Attention

Abstract

While linear attention effectively mitigates the quadratic computational burden associated with softmax attention by employing a recurrent fast-weight state, it frequently underperforms in scenarios involving in-context retrieval and extended context windows. Current solutions have primarily targeted the memory write phase, utilizing techniques such as gating, delta updates, or kernel feature maps, yet they have largely neglected the read phase. In standard linear attention, every stored key adds to the output, causing relevant information to be diluted by the vast majority of irrelevant vectors. To address this, we introduce a computationally efficient read-time query contraction inspired by the geometry of softmax. By applying a second-order Taylor expansion to the softmax log-partition at the isotropic-attention point, we derive a local quadratic model where the curvature aligns with the running covariance of the keys. This covariance can be tracked using the same recurrent or chunkwise mechanisms employed by linear-attention states. The resulting linear operator serves to contract the query along high-density memory directions prior to the state read. We term this approach Curvature-Conditioned Query (CCQ). Since CCQ alters only the read mechanism, it is fully compatible with any linear-attention backbone. When integrated with models like GLA and Gated DeltaNet, it yields enhanced perplexity, improved zero-shot downstream accuracy, better S-NIAH retrieval performance at and beyond the training context length, superior length-extrapolation perplexity (from 4K to 20K), and higher LongBench accuracy, all while incurring minimal additional cost.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC