Title: From Context to Skills: Can Language Models Learn from Context Skillfully?

Abstract:

In many practical applications, language models (LMs) must reason through intricate contexts that surpass their internal parametric knowledge. This necessity gives rise to context learning, a process where LMs acquire relevant information directly from the provided context. A straightforward approach to this problem is inference-time skill augmentation, which involves extracting rules and procedures from the context and converting them into natural-language skills. However, implementing this approach for context learning presents two significant hurdles: the exorbitant expense of manually annotating skills for lengthy, technically dense contexts, and the absence of external feedback to guide automated skill construction.

To address these issues, we introduce Ctx2Skill, a self-evolving framework capable of autonomously discovering, refining, and selecting context-specific skills without requiring human supervision or external feedback. The framework’s foundation is a multi-agent self-play loop comprising three key components: a Challenger, which generates probing tasks and rubrics; a Reasoner, which attempts to solve these tasks using an evolving skill set; and a neutral Judge, which delivers binary feedback.

A critical feature of this system is the evolution of both the Challenger and the Reasoner through accumulated skills. Dedicated Proposer and Generator agents analyze failure cases and synthesize them into targeted skill updates for both parties, facilitating automated skill discovery and refinement. To mitigate the risk of adversarial collapse—resulting from increasingly extreme task generation and over-specialized skill accumulation—we introduce a Cross-time Replay mechanism. This mechanism identifies the skill set that achieves the optimal balance across representative cases for the Reasoner, thereby ensuring robust and generalizable skill evolution.

The skills generated by this process can be integrated into any language model to enhance its context learning capabilities. Evaluations on four context learning tasks from CL-bench demonstrate that Ctx2Skill consistently improves solving rates across various backbone models.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC