Title: Enhancing Visual Speech Recognition Robustness via Head-Pose-Aware FiLM Modulation

Abstract:

Visual Speech Recognition (VSR), which deciphers speech from visual indicators like lip articulation, faces inherent performance bottlenecks due to viseme ambiguity and pose-dependent variations. These factors often result in geometric distortions and occlusions. Current methodologies typically depend on linguistic context or implicit invariance mechanisms, which fail to provide sufficient robustness for visual representations when subjects are not facing the camera directly. To address this limitation, we introduce HP-VSR-ResFiLM, a novel framework that explicitly integrates head-pose data into the visual feature extraction process at the phoneme level.

The proposed architecture employs a two-stage pipeline. Stage 2 utilizes a pretrained NLLB language model to reconstruct text from phonemes, while Stage 1 features a pose-conditioned visual encoder. Specifically, following the 2D CNN frontend, Stage 1 embeds a pose-conditioned residual Feature-wise Linear Modulation (FiLM) block. This component adaptively refines visual features by leveraging head-pose information.

We evaluated HP-VSR-ResFiLM on the LRS2 and LRS3 datasets. Under comparable training conditions and without the need for additional training data, the model achieved word error rates (WER) of 25.0% and 33.2%, respectively, demonstrating competitive performance. Furthermore, ablation studies reveal that incorporating a single residual FiLM block yields consistent improvements in overall WER. Notably, deeper modulation applied at Layers 3 and 4 results in significant performance gains for samples exhibiting yaw angles exceeding 30{\deg}, without compromising accuracy for instances with minor pose variations. These results highlight that explicit, pose-aware feature modulation serves as a computationally efficient and effective strategy for enhancing VSR robustness in unconstrained environments.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC