Title: Threshold-Based Exclusive Batching for LLM Inference

Abstract:

Mixed batching (MB), which interleaves prefill and decode operations within a single batch, has emerged as the prevailing scheduling strategy for large language model (LLM) inference, primarily because it maximizes both memory and compute utilization. However, our controlled experiments reveal that interference between prefill and decode tasks causes the per-step marginal cost of MB to exceed that of pure decode. The point at which this occurs is heavily dependent on hardware bandwidth. On high-bandwidth devices like the H200 (4.8 TB/s), this penalty only manifests when decode tokens constitute more than 80% of the batch. In contrast, on bandwidth-limited hardware such as the RTX PRO 6000 (1.792 TB/s), this threshold drops sharply to just 20%.

These findings indicate that the decision between MB and exclusive batching (EB) is fundamentally governed by GPU memory bandwidth, model dimensions, and workload characteristics. To address this, we establish a closed-form condition defining the performance crossover between EB and MB, alongside asymptotically optimal phase-switching thresholds and memory-safe batch sizing strategies for EB. Leveraging these insights, optimized EB delivers up to 41.9% greater throughput on bandwidth-constrained GPUs, whereas MB remains superior on high-bandwidth systems, particularly with larger models.

We further propose EB+, a hybrid scheduler that implements this crossover condition in real-time to automatically alternate between EB and MB without human oversight. In scenarios involving non-stationary traffic, including shifts in distribution or concurrency, EB+ consistently achieves the highest or near-highest throughput across all tested environments, surpassing MB by as much as 36.4%.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC