Title: vLLM Semantic Router: Signal-Driven Decision Routing for Mixture-of-Modality Models

Abstract:

As large language models expand across various modalities, capabilities, and pricing tiers, the challenge of intelligent request routing—specifically, selecting the optimal model for each query during inference—has emerged as a pivotal systems engineering hurdle. This paper introduces vLLM Semantic Router, a signal-driven framework designed for routing decisions in Mixture-of-Modality (MoM) model deployments.

The core innovation lies in its composable signal orchestration mechanism. The system harvests diverse signal types from incoming requests, ranging from sub-millisecond heuristic features—such as keyword patterns, language identification, context length, and role-based authorization—to neural classifiers assessing domain, embedding similarity, factual grounding, and modality. These heterogeneous signals are combined via configurable Boolean decision rules to establish routing policies tailored to specific deployments.

This architecture allows various deployment scenarios, including multi-cloud enterprise, privacy-compliant, cost-efficient, and latency-critical environments, to be defined through distinct signal-decision configurations without requiring code modifications. The resulting decisions drive semantic model routing, utilizing over a dozen selection algorithms to identify the most cost-effective model based on request characteristics. Additionally, per-decision plugin chains enforce privacy and safety protocols, incorporating jailbreak detection, PII filtering, and hallucination detection through the three-stage HaluGate pipeline.

vLLM Semantic Router offers OpenAI API compatibility for stateful multi-turn interactions and supports multi-endpoint, multi-provider routing across heterogeneous backends, including vLLM, OpenAI, Anthropic, Azure, Bedrock, Gemini, and Vertex AI. It also features a pluggable authorization factory compatible with multiple authentication providers. Implemented in production as an Envoy external processor, the architecture proves that composable signal orchestration allows a single routing framework to effectively manage diverse deployment requirements with varied cost, privacy, and safety policies.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC