Title: Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation

Abstract:

While audio-driven talking-head generation has seen rapid progress, current evaluation standards predominantly depend on frame-wise metrics that presuppose rigid temporal correspondence between output and reference videos. This premise is flawed, as speech-driven facial movements inherently involve slight temporal shifts, variable speaking rates, and stylistic nuances. Consequently, traditional metrics often misinterpret benign timing discrepancies as quality defects, complicating fair comparisons and obscuring the trade-offs inherent in different methods. We propose that evaluating dynamic generative models should be approached as a sequence-alignment task rather than through independent frame comparisons. To this end, we present a unified sequence-level reformulation that incorporates Soft Dynamic Time Warping into standard evaluation workflows. By aligning feature trajectories while maintaining their temporal sequence, our framework offers robustness against limited temporal misalignments without modifying the underlying perceptual, identity, or synchronization encoders. Our analysis demonstrates that frame-wise evaluation is merely a special case of rigid alignment, whereas sequence-level alignment yields greater stability, reduced sensitivity to timing variations, and a clearer distinction between modeling paradigms. Leveraging this principled approach, we executed a large-scale benchmark involving 20 methods across seven datasets, covering canonical, in-the-wild, and style-diverse scenarios under standardized protocols. Our extensive experiments indicate that temporally aligned metrics are more resilient to timing differences, deliver consistent results across diverse datasets, and more effectively expose systematic trade-offs between modeling approaches, such as the balance between synchronization and realism or expressiveness and stability.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC