Title: In-Context Learning Without Training: The Implicit Dynamics

Abstract:

A defining characteristic of Large Language Models (LLMs) is their capacity for in-context learning. This phenomenon allows models to identify and adapt to new patterns at inference time simply by observing examples within the prompt, even when those patterns were absent from the training data. Crucially, this learning occurs without any modification to the model’s weights. Despite its prominence, the underlying mechanisms driving this capability remain largely unexplored.

In this study, we demonstrate that the combination of a self-attention layer and a Multi-Layer Perceptron (MLP) within a transformer block enables the implicit adjustment of the MLP’s weights based on contextual information. Through a blend of theoretical analysis and empirical experimentation, we propose that this straightforward mechanism offers insight into the origins of in-context learning capabilities in LLMs, extending beyond what is accounted for during the training phase. Specifically, our findings indicate that a standard forward pass incorporating context is mathematically equivalent to a context-free forward pass where the MLP weights have been modified by a minimal, low-rank update derived from the context.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC