Title: Many-Shot CoT-ICL: Enabling In-Context Learning to Truly Learn

Abstract: Although many-shot in-context learning (ICL) has demonstrated impressive capabilities, existing research on its scaling dynamics has predominantly concentrated on tasks that do not require reasoning. This study shifts the focus to reasoning-based tasks, specifically examining many-shot chain-of-thought in-context learning (CoT-ICL). By evaluating both reasoning and non-reasoning tasks across LLMs designed for and against reasoning, we uncover several unique characteristics of many-shot CoT-ICL. We interpret these observations by conceptualizing many-shot CoT-ICL as in-context test-time learning, rather than merely scaled pattern matching. Based on this perspective, we propose two core principles: first, demonstrations must be easily comprehensible to the target model; second, they should be arranged to facilitate a seamless conceptual progression. Adhering to these guidelines, we introduce Curvilinear Demonstration Selection (CDS), a straightforward ordering technique that achieves a performance improvement of up to 5.42 percentage points on a mathematical task utilizing 64 demonstrations. Ultimately, our findings reposition the long context window not as a simple retrieval buffer, but as a structured curriculum for in-context test-time learning.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC