Title: VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding

Abstract:

Vision-Language Models (VLMs) face significant hurdles in comprehending long-form videos, largely due to the conflict between computational limits and the necessity of extracting data spread across thousands of frames. Current methods typically either distribute frame sampling evenly—which risks discarding crucial details—or identify keyframes in a single pass, leaving no room for correction if initial selections are suboptimal. To address these issues, we introduce VideoBrain, an end-to-end system that allows VLMs to gather visual data through adaptive, learned sampling strategies. This framework utilizes two complementary agents: a CLIP-based agent designed for semantic retrieval throughout the video and a Uniform agent responsible for dense temporal sampling within specific intervals. In contrast to previous agent-driven techniques that depend on text-only Large Language Models (LLMs) to coordinate visual tools, our VLM directly observes frames and evaluates whether the collected information is sufficient. To stop the model from triggering agents indiscriminately in pursuit of higher rewards, we implemented a data classification pipeline and a behavior-aware reward function, which guide the model to invoke agents only when truly advantageous. Our evaluations across four long-video benchmarks show that VideoBrain outperforms the baseline by 3.5% to 9.0%, all while processing 30-40% fewer frames. Additionally, the model demonstrates robust generalization to short-video benchmarks across different datasets. The source code can be accessed at https://github.com/junbo-zou/VideoBrain.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC