Title: Achieving Practical Scaling and Stability in Memory-Efficient LLM Training via Dynamic Sparsity

Dynamic Sparse Training (DST) presents a compelling approach to enhancing both the efficiency of training and inference for deep neural networks. However, our investigation reveals that applying DST to large language models (LLMs) often triggers optimization instability, characterized by significant loss spikes following topology updates. We identify that the conventional application of standard Adam-based optimizers creates a "cold-start" problem for parameters that are regrown during training. This issue results in disproportionately large parameter updates that disrupt the overall training dynamics.

To resolve these challenges, we introduce Sparse Memory-Efficient Training (SMET). This method stabilizes DST by implementing an optimizer warm-up phase and accelerates training progress through learning-rate scaling that accounts for model density. Additionally, SMET significantly lowers memory requirements by restricting the storage of gradients and optimizer states exclusively to active parameters. Our theoretical analysis of update behaviors under the SMET framework confirms its capacity to enhance optimization stability.

Comprehensive experiments demonstrate that SMET facilitates stable, scalable, and memory-efficient sparse pre-training for LLMs. These findings establish sparse training as a viable and practical alternative to traditional dense training methods. The source code for this work is publicly accessible at: https://github.com/QiaoXiao7282/SMET.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC