Title: Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs

Original: arXiv:2606.01620v1 Announce Type: new

Abstract: While video diffusion models have markedly improved the field of portrait video synthesis, their substantial computational requirements often render them unsuitable for interactive use cases. To address this limitation, we introduce a novel framework capable of generating talking portrait videos in a streamable format, conditioned on both speech audio and reference images. Tailored specifically for streaming environments, the proposed system incorporates an autoregressive latent denoising model alongside a causal video Variational Autoencoder (VAE) designed for deep latent compression. A key innovation of our causal VAE is its ability to ingest a variable quantity of reference images as guidance signals. This mechanism directs the network to prioritize dynamic content over static visual features, which boosts both compression efficiency and reconstruction fidelity. Moreover, we adapt the residual auto-encoding paradigm to better manage spatial-temporal causality within the VAE structure. The generation component utilizes a Rectified Flow Transformer, which outputs video latents through a blockwise auto-regressive process. Our approach facilitates the real-time creation of high-fidelity talking portraits, operating at speeds that significantly surpass baseline models. Extensive experimental results confirm that our method matches or exceeds the performance of these larger models in terms of realism, vividness, and overall video quality.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC