Title: Advancing 3D-Aware Video Diffusion: A Render-Free Approach to Human Motion Control via Mesh Tokenization

Abstract:

While diffusion models have achieved significant milestones in video synthesis, it remains unclear whether they genuinely understand the 3D geometry beneath visual inputs or merely replicate convincing 2D projections. To address this uncertainty, our study examines human motion control—a domain demanding accurate modeling of 3D human anatomy, movement, camera angles, and environmental context. Departing from previous techniques that depend on rendered 2D motion videos for guidance, we introduce a novel framework that bypasses rendering entirely. Instead, video generation is conditioned directly on compressed 3D human mesh tokens. This approach retains comprehensive 3D geometric details and facilitates a cohesive token-based generation pipeline, where video and motion tokens are processed together within a DiT-based architecture. By forcing the model to simultaneously analyze appearance, 3D structure, and viewpoint, this design enhances spatial reasoning. Our experiments reveal superior performance on human motion control benchmarks, notably minimizing artifacts associated with view-dependent 2D guidance and resolving trajectory-pose discrepancies during editing. These results indicate that integrating mesh tokenization into video diffusion models enables a more robust capture of intricate 3D human forms and their dynamic interactions with the surrounding world.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC