Title: ForesightKV: Leveraging Long-Term Impact to Optimize KV Cache Eviction in Reasoning Models

Abstract:

While large language models (LLMs) have demonstrated impressive reasoning capabilities through the generation of extended thought processes, this approach leads to a linear expansion of the key-value (KV) cache as sequence length increases, thereby imposing substantial computational and memory burdens. Current strategies for KV cache eviction attempt to alleviate these costs by removing less critical KV pairs; however, they frequently overlook intricate KV dependencies, which can degrade model performance. To achieve a superior balance between efficiency and accuracy, we present ForesightKV, a training-driven framework designed to predict and execute KV pair eviction during the generation of long texts.

Our methodology begins with the "Golden Eviction" algorithm, which determines the optimal KV pairs to discard at each step by analyzing future attention scores. These decision traces and associated scores are then distilled through supervised training utilizing a Pairwise Ranking Loss. Additionally, we model cache eviction as a Markov Decision Process and employ the GRPO algorithm to counteract the sharp rise in language modeling loss typically observed on low-entropy tokens. Evaluations on the AIME2024 and AIME2025 benchmarks across three reasoning models reveal that ForesightKV consistently surpasses existing methods when operating with only half the standard cache budget. The framework demonstrates synergistic benefits by integrating both supervised learning and reinforcement learning techniques. The source code is accessible at https://github.com/RUCAIBox/ForesightKV.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC