Title: Integrating Model Parallelism with Subnetwork Data Parallelism

Abstract: The pre-training of large-scale neural networks places significant memory burdens on accelerators and frequently necessitates expensive communication overheads. To address these challenges, we present Subnetwork Data Parallelism (SDP), a distributed training framework that divides models into structured subnetworks for training across multiple workers, eliminating the need to exchange activations. Our research examines two distinct masking strategies: backward masking, which enforces sparsity solely during the backward pass to preserve unbiased gradients, and forward masking, which extends sparsity to the forward pass. The latter offers superior efficiency improvements and introduces additional regularization benefits. Additionally, we investigate subnetwork construction at both the neuron and block levels, applying these methods to Transformer and Convolutional Neural Network (CNN) architectures. Empirical results, ranging from pre-training a 1B-parameter LLaMA model on the FineWeb dataset to training ResNet-18 on CIFAR, demonstrate that SDP decreases per-device memory consumption by 28% to 60%. These memory reductions are achieved without compromising performance, which remains stable or improves under FLOP-matched conditions.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC