Title: DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data

Abstract:

Adapting large language models (LLMs) in settings characterized by limited resources and high privacy requirements presents significant hurdles. Because training datasets are frequently fragmented across various clients, decentralized fine-tuning emerges as a viable strategy for collaborative model adaptation without necessitating a central authority. Nevertheless, implementing full-parameter fine-tuning (FPFT) within a decentralized architecture is notoriously difficult. While FPFT delivers robust adaptation capabilities, it demands excessive computational resources for models comprising billions of parameters. Consequently, current decentralized approaches for LLM fine-tuning predominantly utilize parameter-efficient updates. Although these methods enhance efficiency, they can potentially compromise performance on downstream tasks. Furthermore, the prevalence of non-IID (non-independent and identically distributed) data among clients exacerbates the risks of client drift and convergence instability in decentralized optimization processes.

To overcome these obstacles, we introduce DECA, a framework designed for resource-efficient, decentralized full-parameter fine-tuning of LLMs operating on non-IID data. DECA functions by dividing model parameters into separate, non-overlapping blocks and executing sequential block-wise Adam optimization. This approach significantly lowers resource demands while maintaining the benefits of decentralized full-parameter adaptation. To ensure training stability, DECA incorporates first- and second-order block-wise moment estimates that leverage fresh local gradient statistics alongside discrepancy signals derived from consensus mechanisms. Through comprehensive theoretical analysis and extensive empirical evaluations, we demonstrate that DECA delivers rapid convergence, superior downstream performance, and marked improvements in resource efficiency.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC