Title: AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Abstract: Audio-visual speaker tracking seeks to identify and follow active speakers by integrating both auditory and visual information, thereby facilitating detailed, human-focused scene comprehension. This functionality is critical for practical applications including intelligent video editing, surveillance systems, and human-computer interaction. Nevertheless, current datasets predominantly feature simplified or uniform audio-visual environments with imprecise annotations. These overly basic settings skew evaluation metrics toward static audio-visual co-occurrence, failing to thoroughly test robust spatiotemporal modeling and cross-modal reasoning within intricate, dynamic contexts. To overcome these constraints, we present AVTrack, a human-centric audio-visual instance segmentation (AVIS) dataset tailored for dynamic, real-world situations. AVTrack encompasses a wide range of difficult conditions, such as camera movement, visual obstructions, and shifts in position. Tests of leading AVIS methods on the AVTrack dataset demonstrate significant drops in performance, confirming AVTrack’s role as a rigorous benchmark for assessing robust human-centric audio-visual understanding in complex settings. Additionally, we offer a straightforward yet effective baseline to support subsequent research. Project website: https://FudanCVL.github.io/AVTrack/

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC