Title: Pixel Cube: Achieving Photorealistic Portrait Video Relighting via Diffusion Models and Accurate Lighting Simulation

Abstract:

This paper introduces a novel diffusion-based approach for relighting dynamic portrait videos, ensuring both photorealistic quality and temporal stability. The methodology is driven by a unique hybrid training dataset comprising both real-world captured and computationally rendered dynamic portrait videos. This dataset encompasses a wide variety of subject appearances, facial movements, head orientations, and explicit lighting scenarios. To facilitate this, we engineered a specialized LED-based lighting rig designed to emulate realistic lighting conditions while enabling the high-speed acquisition of relighting data.

By capitalizing on the inherent image priors within pre-trained video diffusion models, we developed a high-performance generative framework. This framework utilizes per-frame high dynamic range (HDR) environment maps as primary lighting controls to achieve realistic, identity-preserving relighting. Furthermore, the model incorporates a synthesized background image, providing additional control over the camera’s color tone and exposure levels. The resulting output is a temporally consistent relit video that appears natural and harmonious within the new lighting environment. Crucially, the system faithfully retains the subject’s expressions and intricate facial details, such as skin texture, wrinkles, and facial hair.

The model demonstrates strong generalization capabilities across unseen data, effectively handling variations in subject appearance, motion patterns, and lighting conditions. We conducted extensive experiments involving relighting in-the-wild videos using diverse environment maps and showcased practical applications in portrait photography. Our results indicate that this method sets a new state-of-the-art standard in terms of photorealism, lighting harmony, and temporal consistency.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC