Title: AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations

Abstract:

A primary hurdle in interpretability research continues to be the comprehension of internal representations within large language models (LLMs). While sparse autoencoders (SAEs) present a viable path forward by breaking down activations into understandable features, current methodologies are limited by static sparsity constraints that do not consider the varying complexity of inputs. To address this, we introduce AdaptiveK SAE (Adaptive Top K Sparse Autoencoders), an innovative framework that modulates sparsity levels dynamically according to the semantic complexity of the input data. By employing linear probes, we establish that context complexity is linearly embedded within LLM representations, utilizing this insight to direct feature allocation throughout the training process. Our evaluation across ten distinct language models reveals that this complexity-aware adaptation surpasses fixed-sparsity methods in terms of reconstruction fidelity, explained variance, cosine similarity, and interpretability metrics, all while removing the need for rigorous hyperparameter optimization. The source code is publicly accessible at: https://github.com/hiyukie/adaptiveK.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC