Title: Observation Over Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving

Abstract:

Large Language Model (LLM)-based agents accomplish user objectives via numerous dependent inference steps and tool interactions, generating workloads with total costs that remain unknown at the point of arrival. Current multi-turn architectures utilize the individual turn as the scheduling granularity, determining on a turn-by-turn basis whether to separate the prefill and decode stages. This approach relies on predicting decode length, tool behavior, and KV cache growth—metrics that are unavailable when the scheduler must make decisions. We demonstrate that this reliance on prediction is a constraint of the scheduling unit rather than the workload itself. By elevating the scheduling unit from the turn to the entire conversation, the inherent irregularity of individual turns transforms into a predictable, two-phase structure: an initial compute-bound prefill phase followed by a prolonged, memory-bound tail. Consequently, when the conversation serves as the scheduling unit, resource placement depends solely on the first-turn input length and per-decoder KV occupancy, both of which are directly observable. We implement this concept in ConServe, a system that directs the first-turn prefill to a high-throughput prefiller, moves the KV cache exactly once, and dedicates a single decoder to handle the conversation’s entire tail without employing a learned model for decode-side costs. Compared to a baseline that predicts per-turn, ConServe cuts the p95 time-to-first-effective-token (the latency until the user sees the first output) by 51.08% and boosts energy efficiency by 7.51%, while maintaining Service Level Objectives (SLOs) and last-turn TBT. Additionally, assigning these two phases to heterogeneous GPU tiers yields an extra 22.75% improvement in energy efficiency.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC