Title: On the Stochastic Convergence of Parallel Asynchronous Adaptive First-Order Methods

Abstract: This paper introduces a novel category of asynchronous adaptive first-order optimization techniques, which includes asynchronous adaptations of several widely used algorithms. The study also examines versions of these methods that incorporate momentum and/or inexact normalization. We analyze the convergence behavior of this class of methods when applied to non-convex functions within a fully stochastic framework. Under plausible assumptions, the convergence rate is demonstrated to be O(1/sqrt{t}), up to logarithmic factors. Empirical results indicate that these asynchronous adaptive algorithms are highly suitable for heterogeneous, large-scale machine learning environments.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC