Title: Mastering Rigid Body Dynamics: A Study on Action-Conditional, Object-Centric Gaussian Splatting World Models

World models serve as critical tools for intelligent agents, allowing them to forecast the outcomes of their interventions within an environment. In this study, we introduce the Multi Rigid Object Gaussian World Model (MRO-GWM), an innovative framework designed to capture the action-dependent dynamics of rigid bodies in three-dimensional space. By utilizing object-centric Gaussian representations, our approach facilitates the accurate modeling of complex scenes featuring multiple objects with arbitrary geometries.

The core of our architecture is a specialized spatio-temporal transformer that leverages historical data of object Gaussians alongside anticipated future actions to predict subsequent rigid body movements. We encode objects within a canonical frame using Gaussians, a method that simplifies the description of motion as rigid body transformations. Since the model is trained on multi-viewpoint reconstructions, it is inherently equipped to manage partial observations caused by occlusions.

We rigorously assess the predictive capabilities of our method using synthetic datasets that simulate multi-object dynamics and interactions driven by a robot end-effector, featuring common household items. Furthermore, we demonstrate the practical utility of MRO-GWM by applying it to model-predictive control tasks for non-prehensile manipulation within a simulation environment.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC