Title: Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention

Abstract:

Implicit in-context learning (ICL) has recently surfaced as a promising framework that emulates ICL behaviors within the representation space of large language models (LLMs). This approach seeks to achieve few-shot performance capabilities at the computational cost of zero-shot inference. Nevertheless, current methodologies predominantly depend on injecting shift vectors into residual flows—vectors typically derived from labeled demonstrations or task-specific alignment. These designs fail to leverage the underlying structural mechanisms of ICL and consequently exhibit limited generalizability.

To overcome these limitations, we introduce In-Context Routing (ICR), an innovative implicit ICL technique that identifies and exploits generalizable ICL patterns at the level of attention logits. ICR extracts reusable structural directions that arise during the ICL process and utilizes a learnable, input-conditioned router to adjust attention logits. This mechanism facilitates an efficient "train-once-and-reuse" paradigm. We assessed ICR across 12 real-world datasets that cover various domains and different LLMs. Our findings indicate that ICR consistently surpasses existing implicit ICL methods that necessitate task-specific retrieval or training, while also showing robust generalization to out-of-domain tasks where those methods typically falter. These results suggest that ICR is poised to expand the practical utility of ICL. The code is available at https://github.com/Lijiaqian1/In-Context-Routing.git.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC