Title: AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

Original: arXiv:2605.03644v2 Announce Type: replace

Abstract:

Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs). However, existing methods typically rely on a predetermined, fixed number of shots. This static approach often fails to adapt to the varying difficulty of different queries, leading to either insufficient context or interference from noise. Furthermore, the prohibitive computational and memory costs of long contexts severely limit Many-Shot's feasibility. To address the above limitations, we propose AdapShot, which dynamically optimizes shot counts and leverages KV cache reuse for efficient inference. Specifically, we design a probe-based evaluation mechanism that utilizes output entropy to determine the optimal number of shots. To bypass the redundant prefilling computation during both the probing and inference phases, we incorporate a semantics-aware KV cache reuse strategy. Within this reuse strategy, to address positional encoding incompatibilities, we introduce a decoupling and re-encoding method that enables the flexible reordering of cached key-value pairs. Extensive experiments demonstrate that AdapShot achieves an average performance gain of around 10% and a 4.64x speedup compared to state-of-the-art DBSA.

Rewritten:

Title: AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

Original: arXiv:2605.03644v2 Announce Type: replace

Abstract:

The use of Large Language Models (LLMs) for Many-Shot In-Context Learning (ICL) represents a promising avenue for enhancing reasoning capabilities through the utilization of numerous examples. Nevertheless, current approaches generally depend on a static, pre-set number of shots. This rigidity proves inadequate for handling queries of varying complexity, frequently resulting in either a lack of necessary context or the introduction of disruptive noise. Additionally, the substantial memory and computational demands associated with processing long contexts pose significant barriers to the practical application of Many-Shot techniques. To overcome these challenges, we introduce AdapShot, a framework that dynamically adjusts the number of shots and employs KV cache reuse to streamline inference. Our method features a probing-based assessment tool that calculates output entropy to identify the ideal shot count. By integrating a semantics-aware KV cache reuse strategy, we eliminate the need for redundant prefilling calculations during both the probing and inference stages. To resolve issues related to positional encoding mismatches within this reuse framework, we propose a decoupling and re-encoding technique that allows for the flexible rearrangement of stored key-value pairs. Our comprehensive experiments reveal that AdapShot delivers an average performance improvement of approximately 10% and accelerates processing by 4.64 times relative to the leading DBSA method.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC