Title: Achieving Robust Self-Supervised Driver Distraction Detection Through Multi-Modal Video Representation Alignment

Abstract: The development of robust self-supervised learning for multi-modal video representations is essential for practical applications like driver distraction detection, a field where diverse sensors yield complementary yet noisy data streams. Traditional contrastive loss functions, including InfoNCE, operate on the premise that all negative samples are equally informative and all positive samples are accurate. However, this assumption often breaks down in multi-modal contexts due to factors such as semantic overlap between modalities, viewpoint variations, and occlusions. To tackle these issues, we present a novel framework for multi-modal global alignment that simultaneously accounts for faulty negatives and unreliable or incorrect positives. Our method employs a weighting mechanism grounded in similarity distributions to reduce the influence of noisy positives, while utilizing cycle-consistency scores to generate soft targets, thereby alleviating the rigid hard-negative constraint. This approach expands standard pairwise alignment into a principled global multi-modal context by aggregating alignment data across all possible modality pairs. We tested our method on the Drive&Act dataset, where it consistently surpassed both pairwise and established global alignment baselines across RGB, IR, Depth, and Skeleton modalities. Furthermore, cross-view ablation studies confirmed the strong generalization capabilities of our representations to unseen camera angles, underscoring their robustness. Ultimately, this framework offers a scalable and effective solution for self-supervised global multi-modal representation learning, facilitating reliable driver distraction detection and advancing real-world multi-modal video understanding. Our code will be made available on GitHub.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC