Title: MuLoCo: Demonstrating Muon as a Viable Inner Optimizer for DiLoCo

Abstract:

DiLoCo serves as a robust framework for training large language models (LLMs), facilitating larger optimal batch sizes and improved accelerator efficiency even when constrained by network bandwidth. However, prior research (Charles et al., 2025) indicates that DiLoCo’s efficacy diminishes as the worker count ($K$) rises. This study argues that a critical, yet frequently ignored, determinant of DiLoCo’s performance is the selection of the inner optimizer, which directly influences the pseudogradient utilized by the outer optimizer.

Motivated by Muon’s recent superiority over AdamW in data parallel (DP) training contexts, we investigate how Muon’s normalized optimization steps impact the quality of the pseudogradient. Our analysis reveals that, compared to AdamW, Muon generates pseudogradients that are more directionally accurate as the number of workers ($K$) grows.

To validate these findings, we performed extensive hyperparameter tuning across model scales of 150M, 416M, 914M, 1.76B, and 3.1B. We compared four configurations: DiLoCo, MuLoCo (DiLoCo with Muon as the inner optimizer), AdamW DP, and Muon DP. The results demonstrate that MuLoCo consistently surpasses standard DiLoCo in absolute performance metrics for all $K\geq1$. Furthermore, for $K>2$, MuLoCo exhibits better relative performance compared to its data parallel counterparts. Additionally, MuLoCo remains compatible with quantization, streaming, and extended synchronization intervals.

Notably, at $K=1$, MuLoCo not only exceeds the performance of the data-parallel baseline but also supports larger critical batch sizes. We further extrapolated optimal hyperparameters to the 15B scale and executed training runs using six distinct configurations with $K=1$ and $K=16$ workers. At this scale, MuLoCo with $K=16$ workers nearly replicates the performance achieved by a single worker. Conversely, MuLoCo with $K=1$ matches the highest-performing baseline while utilizing a significantly larger batch size of 16M tokens.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC