Title: ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Original: arXiv:2606.00024v1 Announce Type: new Abstract: Long-context decoding in Large Language Models (LLMs) is severely constrained by the memory bandwidth required to fetch the extensive Key-Value (KV) cache. Most existing KV management methods rely on key-only pruning before decoding, despite the evidence that attention outputs depend jointly on keys and values, as incorporating values in their methods incurs prohibitive additional overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. This design makes ART orthogonal to existing key-based KV cache management methods, enabling seamless integration with them. Experiments on LongBench benchmarks show that ART achieves 20% higher generation throughput in large batch size than state-of-the-art baseline while maintaining comparable accuracy.

Rewritten:

Title: ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Abstract: The decoding of long contexts in Large Language Models (LLMs) faces significant bottlenecks due to the substantial memory bandwidth needed to retrieve the extensive Key-Value (KV) cache. While current KV management strategies typically employ pruning based solely on keys prior to the decoding phase, this approach overlooks the fact that attention outputs are influenced by both keys and values. Integrating values into existing methods is generally avoided because it introduces excessive computational overhead. To address this, we introduce Attention Run-time Termination (ART), a lightweight mechanism that monitors accumulated attention outputs in real-time during kernel execution. ART halts further access to KV blocks once their additional impact becomes insignificant. Because ART operates independently of key-based cache management techniques, it is fully compatible and can be easily integrated with current solutions. Our evaluations on the LongBench benchmarks demonstrate that ART boosts generation throughput by 20% compared to state-of-the-art baselines when using large batch sizes, all while preserving similar levels of accuracy.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC