Title: Exploring Design Space for Finer-Grain Compute-Communication Overlap via DMA

Abstract: Current machine learning workloads necessitate distributing both training and inference processes across multiple GPUs. Despite this, parallelization strategies frequently encounter exposed critical-path communication bottlenecks, resulting in a missed opportunity for a potential 1.7x performance gain through the overlap of computation and data transfer. Existing methods for achieving this overlap leverage the fact that ML model states and inputs are already partitioned across GPUs, aligning computation and communication at the shard level. However, this coarse-grained approach is constrained by restricted network topology compatibility and inefficient dataflows.

To address these limitations, we advocate for a finer-grain compute-communication overlap strategy, which we call FiCCO. Operating at a granular level deeper than traditional sharding, FiCCO facilitates overlap across a broader range of network topologies and supports more refined dataflow patterns. Our analysis demonstrates that FiCCO expands the feasible design space of execution schedules beyond what is achievable with shard-level techniques alone. To navigate this expanded space, we investigate and characterize the performance inefficiencies associated with overlap, mapping schedules to their specific inefficiency profiles.

Our characterization identifies decomposition delays and resource contention as the primary performance constraints. We establish a correlation between these slowdown factors and the static sizes of compute and communication operators. This insight enables the development of heuristics—designed for integration into frameworks and runtimes—that can select customized FiCCO schedules tailored to the specific characteristics of underlying ML operations. Furthermore, to mitigate the contention inefficiencies inherent in operation overlap, we offload communication tasks to GPU Direct Memory Access (DMA) engines.

Evaluations across various scenarios derived from realistic ML deployments show that our heuristic-driven, bespoke schedules achieve speedups of up to 1.6x. Additionally, our heuristics demonstrate high accuracy, correctly identifying the optimal schedule in 81% of previously unseen scenarios.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC