Title: Bridging Concrete and Abstract Reasoning: The Synergy Between World Models and Language Models

Abstract:

Multimodal Large Language Models (MLLMs) and world models offer distinct, complementary strengths in forecasting future events based on static visual inputs. While world models excel at generating specific, concrete visual projections of potential futures, MLLMs are adept at performing abstract reasoning regarding objectives, rules, and queries. However, these visual simulations are inherently stochastic; they may appear visually realistic yet fail to align with task requirements. This discrepancy necessitates a mechanism to assess the utility of visual simulation, validate the credibility of generated rollouts, and determine how such simulations should inform the final decision.

We address this challenge through the framework of "controlled concrete reasoning," a process where a model learns to selectively invoke, verify, and merge visual future simulations with abstract logic. To evaluate this approach, we introduce two benchmarks verified by human experts: VRQABench, designed for controllable spatial lookahead, and OpenWorldQA, which focuses on open-domain physical prediction. Additionally, we propose a training method termed Privileged-Future On-Policy Self-Distillation (PF-OPSD). During the training phase, PF-OPSD leverages ground-truth future videos and answers as privileged context on the teacher side to assess the on-policy trajectories of concrete reasoning. Crucially, the deployable student model is not exposed to true future data during inference.

Our experiments demonstrate that PF-OPSD surpasses baseline methods by 10.6% on VRQABench and 10.9% on OpenWorldQA, while also enhancing robustness against noisy or contradictory rollouts. The codebase and datasets are publicly accessible at https://github.com/yczhou001/PF-OPSD.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC