Title: Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution

Abstract:

While sparse large language models (LLMs) present a compelling path toward efficient deployment, tailoring them for specific downstream tasks proves difficult. The primary challenge lies in facilitating effective task adaptation without compromising the inherent efficiency gains provided by sparsity. Current fine-tuning approaches are ill-equipped for this environment; they typically either add dense parameters or rely on a static sparse structure, thereby restricting their compatibility with sparse LLMs. To address these limitations, we introduce Sparsity Evolution Fine-Tuning (SEFT), a framework purpose-built for sparse LLMs. SEFT enables the sparse architecture to evolve throughout the fine-tuning process by periodically redistributing task-specific sparse updates and reactivating weights that were previously pruned if it proves advantageous. Concurrently, the framework maintains the efficiency benefits of sparsity by employing topology adaptation driven by parameter importance. Evaluations across various benchmarks using LLaMA, DeepSeek, and Mistral models demonstrate that SEFT achieves superior performance alongside enhanced memory and time efficiency relative to existing baseline methods. The code for this work is publicly accessible at: https://github.com/QiaoXiao7282/SEFT.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC