Title: StreamingVLM: Enabling Real-Time Comprehension for Continuous Video Feeds

Abstract:

While Vision-Language Models (VLMs) hold significant promise for driving autonomous agents and real-time assistants, they encounter a major hurdle: processing near-endless video streams without incurring excessive memory demands or latency. Traditional methods that apply full attention to entire videos result in quadratic computational overhead and degrade performance on extended content. Conversely, basic sliding window techniques are inadequate, as they either disrupt narrative coherence or incur high latency through redundant calculations.

To address these issues, we present StreamingVLM, a framework engineered for stable, real-time comprehension of infinite visual inputs. Our solution unifies training and streaming inference. During operation, we preserve a lean Key-Value (KV) cache by leveraging attention sinks, a short buffer of recent visual tokens, and a longer buffer of recent textual tokens. This streaming capability is achieved through a straightforward Supervised Fine-Tuning (SFT) approach. This strategy applies full attention to short, overlapping video segments, effectively replicating the attention patterns used during inference without requiring training on prohibitively long contexts.

For assessment, we developed Inf-Streams-Eval, a novel benchmark featuring videos that average over two hours in length, necessitating dense, per-second synchronization between visual frames and text. In this benchmark, StreamingVLM secured a 66.18% win rate against GPT-4O mini. The model delivers consistent, real-time performance, reaching up to 8 frames per second (FPS) on a single NVIDIA H100 GPU. Remarkably, this SFT methodology also boosts general Visual Question Answering (VQA) capabilities without needing VQA-specific fine-tuning, yielding improvements of +4.30 on LongVideoBench and +5.96 on OVOBench Realtime. The source code is accessible at https://github.com/mit-han-lab/streaming-vlm.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC