Title: Heterogeneous Decentralized Diffusion Models

Abstract:

The training of frontier-scale diffusion models typically demands immense computational power, confining participation to well-funded institutions due to the need for tightly coupled clusters. Although Decentralized Diffusion Models (DDM) facilitate the isolated training of multiple experts, current methods are constrained by homogeneous training objectives and high resource costs, such as the 1176 GPU-days required by existing approaches. We introduce a streamlined framework that significantly lowers these barriers while accommodating heterogeneous training goals. Our solution integrates three primary innovations: first, a decentralized training paradigm that permits experts to utilize distinct objectives—specifically DDPM and Flow Matching—which are unified during inference without the need for retraining; second, a conversion method for pretrained checkpoints from ImageNet-DDPM to Flow Matching objectives, which accelerates convergence and removes the necessity for objective-specific pretraining; and third, the adoption of PixArt-$\alpha$’s efficient AdaLN-Single architecture, which lowers parameter counts without compromising output quality. Evaluations on the LAION-Aesthetics dataset demonstrate that, compared to the training scale of previous DDM studies, our method cuts computational requirements by 16 times and data usage by 14 times. Furthermore, under consistent inference conditions, our heterogeneous setup outperforms the homogeneous baseline in both FID scores and intra-prompt diversity. By removing synchronization dependencies and supporting a mix of DDPM and Flow Matching objectives, our framework democratizes access to decentralized generative model training, allowing contributors with single GPUs possessing just 24–48GB of VRAM to participate.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC