**Title: Activation-Based Active Learning for In-Context Learning: Challenges and Insights

Abstract:

While deep active learning has been previously investigated for selecting in-context samples for Large Language Models (LLMs), prior work has largely overlooked techniques leveraging recent breakthroughs in understanding transformer activations. This study investigates the hypothesis that model activations can offer a granular signal to enhance the selection of in-context examples. We provide the most thorough examination to date of MLP activation-based deep active learning approaches within the context of in-context learning. Our analysis explores the effects of various attention masking strategies on active learning performance across a range of classification and generative datasets, utilizing both Llama-3.2-3B and Qwen2.5-3B base models.

Contrary to expectations, our findings reveal a negative outcome: MLP outputs, when analyzed through the perspective of massive activations or the first four statistical moments, show no meaningful correlation with either example quality or overall task performance. Specifically, the absolute Spearman correlation coefficient never exceeded 0.33 across all tested models and tasks. Consequently, we conclude that activation-based sampling is unsuitable for in-context learning. We attribute this limitation potentially to the phenomenon of superposition, where models encode more features than their dimensional capacity allows. This suggests that alternative methodologies, such as Sparse Autoencoders (SAEs), may represent a more viable direction for future research.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC