Title: Value-Aware Stochastic KV Cache Eviction for Reasoning Models

Abstract:

While reasoning models achieve greater accuracy by employing extended chains of thought, the resulting lengthy outputs impose significant memory and computational constraints. Although Key-Value (KV) cache eviction techniques alleviate these burdens by discarding less critical key-value pairs, they frequently result in lower accuracy compared to selection-based sparse attention methods, which retain the complete KV cache. This study highlights two pivotal elements that determine the success of KV cache eviction in preserving model performance. First, a minor portion of value states exhibit disproportionately high magnitudes; removing these can lead to catastrophic failures, such as models becoming trapped in repetitive reasoning cycles. Second, incorporating stochasticity into the eviction process enhances accuracy by fostering greater diversity within the cache. Drawing on these insights, we introduce Value-aware Stochastic KV Cache Eviction (VaSE), a training-free strategy designed to safeguard high-magnitude value states and encourage varied eviction choices. In evaluations across six reasoning tasks, Qwen3 models utilizing VaSE with 4x KV cache compression achieved superior average accuracy compared to the state-of-the-art selection method at equivalent sparsity levels. Furthermore, VaSE surpassed the most effective eviction approach by over 4%. Ultimately, VaSE effectively narrows the divide between efficiency and precision, supports FlashAttention2, and provides reasoning models with a static memory footprint.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC