Title: Real2SAM2Real: Enhancing Video Diffusion with Generative 3D Caches as Complementary Context

Abstract:

Although Video Diffusion Models (VDMs) are highly capable of producing high-quality video content, achieving accurate control over camera movements and scene composition remains a significant hurdle. Current approaches mostly depend on implicit diffusion priors to reconstruct unseen areas, which frequently results in structural failures when dealing with rapid motion or intricate occlusions. In response to these limitations, we introduce Real2SAM2Real, a novel framework that utilizes 3D lifting models, such as SAM3D, to generate an explicitly editable 3D cache. This cache acts as a sturdy geometric foundation for the VDM. Unlike methods that only capture visible surfaces, our approach secures the complete 3D volume of foreground objects, thereby embedding comprehensive spatial priors into the VDM and offering reliable, 3D-aware guidance for complex scene dynamics.

To harness this 3D guidance effectively without compromising the model’s pre-trained knowledge, we have developed a Soft Spatial-Aligned Injection mechanism combined with a fine-tuning strategy designed to be minimally invasive for VDMs. Additionally, we utilize masked normal maps to facilitate a cross-modal bridge, enabling a data curation and perturbation pipeline that does not require 3D data. Our extensive experimental results show that Real2SAM2Real allows for precise, independent control over both multi-object motions and camera trajectories. By integrating complementary information from generative 3D caches, our framework resolves common issues stemming from an over-dependence on diffusion priors, ensuring robust spatiotemporal consistency even during significant camera shifts and heavy occlusions. Importantly, by separating geometry from visual appearance, our VDM-specific 3D cache eliminates perspective distortions caused by missing structural data, fake surfaces, and confusing signals from reflections or refractions.

The project website can be accessed at https://jiayi-wu-leo.github.io/real2sam2real.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC