Title: SRA: Span Representation Alignment for Large Language Model Distillation

Abstract:

Cross-Tokenizer Knowledge Distillation (CTKD) facilitates the transfer of knowledge from a large language model to a more compact student model, even when the two systems utilize distinct tokenization schemes. While current methodologies predominantly emphasize token-level alignment strategies, these approaches are frequently fragile and highly susceptible to inconsistencies arising from tokenizer discrepancies. We contend that the strategy for aggregating tokens into more stable representations prior to distillation holds equal significance. To address this, we present SRA (Span Representation Alignment for Large Language Model Distillation), an innovative framework that reinterprets CTKD through the perspective of Multi-Particle Dynamical Systems.

SRA transforms the core unit of alignment from individual tokens to robust, tokenizer-independent spans. In this model, each span is treated as a cluster of particles, with its state defined by its Center of Mass (CoM)—an attention-weighted average that encapsulates comprehensive semantic data. By utilizing attention-derived weighting for span centers of mass, the method prioritizes the most significant spans. Furthermore, we incorporate a geometric regularizer to maintain the structural coherence of the representation space and implement aligned span logit distillation to bolster knowledge transfer between models. In rigorous cross-architecture distillation trials, SRA consistently and substantially surpasses existing state-of-the-art CTKD baselines, thereby validating the efficacy of our physically grounded methodology.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC