Title: DeMuon: A Decentralized Muon for Matrix Optimization over Graphs

Abstract:

This study introduces DeMuon, a novel approach designed for decentralized matrix optimization within specified communication topologies. Building upon its centralized counterpart, Muon, DeMuon integrates matrix orthogonalization through Newton-Schulz iterations. Additionally, it utilizes gradient tracking to address the heterogeneity found in local functions. We derive the iteration complexity required for DeMuon to converge to an approximate stochastic stationary point, operating under heavy-tailed noise conditions alongside standard mild assumptions. Notably, this complexity aligns with the most efficient known bounds for centralized algorithms regarding their dependence on target tolerance. To our knowledge, DeMuon represents the inaugural direct adaptation of Muon to decentralized graph-based optimization that offers rigorous complexity guarantees. We also present initial numerical experiments involving the decentralized pretraining of transformers across graphs with diverse connectivity levels. The results indicate that DeMuon consistently outperforms other established decentralized algorithms, showing significant advantages across various network structures.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC