Title: Rethinking Granularity in Replacement-Based LLM Compression: From Layers to Submodules

Abstract

Post-training compression techniques for Large Language Models (LLMs) typically function by eliminating whole architectural components or swapping them out for fitted modules. Current replacement-based approaches are bound by two primary constraints: they operate at the granularity of full layers and select these components contiguously. We contend that this approach is unnecessarily limiting. Evidence suggests that redundancy within pretrained transformers is neither localized to contiguous sections nor uniformly distributed between Attention and FeedForward outputs. This implies that optimal approximation strategies vary by submodule type and that removable elements do not need to be clustered within specific depth ranges.

Guided by this insight, we propose SubFit (Submodule-level Fitted residual replacement), a method that compresses LLMs at the submodule level. In this framework, Attention and FeedForward submodules are chosen non-contiguously, with each assigned its own lightweight fitted residual bypass. Operating entirely post-training, SubFit relies solely on calibration data.

We evaluated SubFit across ten LLMs (comprising five base models and five instruction-tuned variants), four replacement-based baselines, and five sparsity levels ranging from 12.5% to 37.5%. SubFit demonstrated the most favorable aggregate perplexity-accuracy trade-off across all tested sparsity levels, with performance advantages becoming more pronounced under aggressive compression settings. Specifically, at 25% sparsity, SubFit maintained 84.6% of dense downstream accuracy with a perplexity degradation of 2.42x. In contrast, the strongest baseline methods retained only 81.6% of accuracy and suffered a 4.34x perplexity increase. Additionally, SubFit provided measurable improvements in inference speed and KV-cache efficiency. The implementation code is publicly available at https://github.com/eliacunegatti/SubFit.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC