VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization
Title: Enhancing Video Reasoning Through Adaptive Test-Time Optimization with VLMs as Instructors
Abstract:
The emerging "Reasoning with Video" framework employs Video Generation Models (VGMs) to create temporally consistent visual sequences for solving reasoning problems. While state-of-the-art VGMs deliver high-quality visuals, they frequently encounter logical errors in complex reasoning scenarios because they struggle to comprehend and adhere to specific task rules. Previous approaches have attempted to mitigate this by deploying Vision-Language Models (VLMs) as pre-solvers to generate or refine textual instructions for the VGM. However, text-based guidance often lacks the necessary spatiotemporal nuance, and VGMs continue to fail in executing fine-grained or long-tail instructions, even when provided with a sound plan.
Although VLMs are not ideal solvers, they demonstrate exceptional perceptual abilities in assessing whether process constraints are met and final objectives are achieved. Capitalizing on this strength, we propose a paradigm shift that repositions VLMs from solvers to "teachers." In this new framework, a VLM teacher distills task-specific rules into differentiable rewards. These rewards guide a VGM Reasoner through test-time online optimization of a lightweight LoRA module. This approach facilitates adaptive optimization during testing and expands the VGM’s reasoning capacity beyond its inherent limitations.
Our evaluations on both symbolic (VBVR-Bench) and general-purpose (RULER-Bench) video reasoning benchmarks indicate that this method achieves an average performance improvement of 16.7 points. This significantly outperforms the VLM-as-Solver approach (+0.4 points) and Best-of-N scaling (+2.2 points) while maintaining comparable test-time costs. These results suggest that employing VLMs as test-time teachers presents a viable and promising pathway for developing generalizable video reasoning systems.
Project Page: https://VLM-as-Teacher.github.io/
Source: arXiv Generated at: 2026-06-02 00:00:00 UTC
