Title: MSAVBench: Advancing the Comprehensive and Reliable Assessment of Multi-Shot Audio-Video Generation

Original: arXiv:2605.20183v2 Announce Type: replace

Abstract:

As the landscape of video generation shifts from single-shot synthesis toward intricate multi-shot audio-video (MSAV) narratives to satisfy practical requirements, the evaluation of these cutting-edge models has become a significant hurdle. Current benchmarks suffer from narrow scopes and limited data variety, often employing inflexible evaluation protocols that hinder the systematic and trustworthy assessment of contemporary MSAV systems. To address these deficiencies, we present MSAVBench, the inaugural comprehensive benchmark coupled with an adaptive hybrid evaluation framework designed specifically for multi-shot audio-video generation.

This benchmark encompasses four primary dimensions—video, audio, shot, and reference—spanning a wide array of task configurations. It supports varying shot counts of up to 15 and incorporates challenging, non-realistic scenarios. The proposed evaluation framework enhances robustness by integrating an adaptive self-correction mechanism for shot segmentation, instance-specific rubrics for subjective metrics, and tool-based evidence extraction to facilitate complex judgments.

MSAVVBench demonstrates strong concordance with human evaluations, achieving a Spearman rank correlation of 91.5%. Our extensive analysis of 19 state-of-the-art closed- and open-source models reveals that existing systems continue to face difficulties with director-level control and precise audio-visual synchronization. However, modular or agentic generation pipelines appear to offer a viable strategy for closing the performance gap between open- and closed-source models. The benchmark dataset and evaluation code are openly accessible at https://github.com/ali-vilab/MSAVBench.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC