Title: CoSTL: Comprehensive Spatial-Temporal Representation Learning for Moment Retrieval and Highlight Detection

Abstract:

Video Moment Retrieval (MR) and Highlight Detection (HD) are essential components of video analysis, tasked with pinpointing specific segments and assessing the relevance of clips according to a provided text query. While contemporary methods often address these as analogous video grounding problems, employing identical architectures for both, they inherently demand a dual capability: granular comprehension at the image level and sophisticated temporal understanding across the full video duration. Current techniques predominantly rely on frame-level features for temporal modeling, frequently overlooking the detailed visual data within individual frames that aligns with the textual query. This omission often results in suboptimal grounding accuracy.

To overcome this constraint, we introduce the Comprehensive Spatial-Temporal Representation Learning Framework (CoSTL). This approach simultaneously extracts fine-grained spatial details and captures temporal dynamics. CoSTL features a text-driven progressive fine-grained image encoder that utilizes a two-stage process for text-driven knowledge extraction, thereby learning detailed spatial representations. Additionally, a multi-scale temporal perception module is employed to gather comprehensive spatial-temporal features, which significantly boosts the model’s capacity to interpret temporal changes. Our method achieves state-of-the-art results across four established public benchmarks: QVHighlights, Charades-STA, TACoS, and TVSum.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC