Title: NVIDIA OmniDreams: A Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

Abstract

As the capabilities of autonomous vehicles continue to evolve, the safe assessment of driving policies within long-tail scenarios remains a significant hurdle. In closed-loop simulations, the driving policy model engages in active interaction with its environment; its decisions dynamically modify the simulator’s state, which in turn dictates the subsequent sensor observations generated. Although recent neural simulators based on reconstruction techniques provide photorealistic visuals, they are inherently limited by their reliance on initial captured datasets and often fail to generalize effectively to highly dynamic or entirely novel scenes.

To address these constraints, we present OmniDreams, a foundational generative world model that has undergone mid- and post-training based on the Cosmos diffusion model. This architecture is designed to autoregressively produce action-conditioned video in real time. By utilizing the extensive visual priors inherent in Cosmos and fine-tuning on 21,000 hours of driving scenarios, OmniDreams is capable of synthesizing complex, previously unobserved phenomena that traditional simulators struggle to replicate. These include extreme weather conditions and the unpredictable behaviors of dynamic agents.

A key feature of OmniDreams is its ability to autoregressively condition the generation of photorealistic sensor data on past frames, the current state of the simulator, and immediate driving actions. When integrated into a closed-loop system comprising the Alpamayo 1 policy model and the AlpaSim orchestrator, OmniDreams functions as a highly responsive and reactive environment, offering a scalable and comprehensive framework for training and evaluating next-generation autonomous driving policies.

Furthermore, we present preliminary findings demonstrating that a world-action model (WAM) post-trained from OmniDreams delivers robust performance on the Physical AI Autonomous Vehicles NuRec dataset. Notably, this approach outperforms the VLA-based Alpamayo 1.5 research policy model while utilizing only one-fifth of the total parameters. These results underscore the potential for real-time world models like OmniDreams to serve not only as simulation tools but also as foundational backbones for policy architectures.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC