Title: PointAction: 3D Points as Universal Action Representations for Robot Control

Abstract:

Video-Action Models (VAMs) offer a promising trajectory toward generalizable robot manipulation by capitalizing on the extensive visual dynamics learned by pre-trained video diffusion models. However, relying solely on RGB video rollouts presents significant challenges, as these outputs fail to explicitly define metric 3D motion, contact geometry, and fine-grained spatial constraints, thereby creating ambiguity in action grounding. Furthermore, the process of scaling action supervision across various tasks and robot embodiments is prohibitively expensive.

To address these issues, we introduce PointAction, a framework that connects video predictions to robot actions via explicit point-based 4D modeling. By fine-tuning a foundation video generation model, PointAction simultaneously forecasts future RGB frames and dynamic 3D pointmaps. This approach generates temporally consistent 3D motion for the geometry relevant to the task. These dynamic points function as a structured, embodiment-agnostic interface for actions, which is then mapped to executable robot commands by a diffusion-based action decoder.

By employing metric 3D point dynamics as the bridge between video prediction and control, PointAction mitigates the ambiguity inherent in RGB-only action grounding. This method facilitates transfer learning across different tasks and embodiments while requiring minimal action supervision. Our experiments demonstrate that PointAction sets a new state-of-the-art in 4D generation quality for robot scenes, surpasses current baselines in simulation environments, and successfully generalizes to two real-world robot arms that were not part of the pretraining data.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC