Title: DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference

Abstract:

The escalating demand for efficient multi-tenant GPU scheduling has been driven by the swift expansion of large language model (LLM) inference services. Although contemporary inference runtimes, such as vLLM, enhance throughput via continuous batching and refined memory management, precisely estimating the runtime costs of diverse inference requests remains a formidable obstacle. In real-world scenarios, actual output lengths frequently diverge from the estimates made at the time of admission. This phenomenon, known as runtime token drift, can result in workload misclassification, queue imbalances, heightened tail latency, and a subsequent deterioration in Quality-of-Service (QoS).

To address these challenges, this study introduces DriftSched, an adaptive QoS-aware scheduling framework designed for multi-tenant LLM inference on NVIDIA L4 GPUs. DriftSched integrates workload classification, token-budget estimation, tenant-aware queue management, and runtime feedback-driven drift compensation to refine admission-time scheduling decisions. The framework assesses several scheduling policies—specifically FIFO, Priority, Weighted, Shortest-Job-First (SJF), and Aging Priority—across heterogeneous multi-tenant workloads.

Experimental findings reveal significant runtime token drift across various workload categories. By employing adaptive bias correction, the framework reduces workload estimation error by an average of 38.8% in terms of Mean Absolute Error (MAE) and 40.5% in Root Mean Square Error (RMSE). This enhancement leads to greater stability in workload classification and improved scheduling accuracy. Among the schedulers tested, SJF demonstrated superior overall performance, decreasing median end-to-end latency by roughly 42% and P99 latency by approximately 16% compared to FIFO under conditions of sustained GPU contention. This research contributes an adaptive drift-aware scheduling architecture, a mechanism for runtime token-drift compensation, and a reproducible benchmarking framework for evaluating QoS-aware LLM inference scheduling on shared GPU infrastructure.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC