Title: UF-AMA: A Unified Framework for Cross-Domain Emotion Recognition via Adaptive Multimodal Alignment

Abstract:

Recent years have seen a surge in interest regarding emotion recognition systems that rely on physiological indicators, particularly electroencephalogram (EEG) data. This shift is driven by the superior objectivity and reliability of internal physiological metrics compared to external behavioral cues, such as facial expressions. Nevertheless, building robust and highly generalizable cross-domain multimodal emotion recognition models remains a significant hurdle. This difficulty arises from distribution shifts stemming from individual and contextual variances, as well as inconsistencies in sample quality across different modalities.

To tackle the challenges of cross-subject and cross-session emotion recognition using multimodal physiological signals, this study introduces the Unified Framework with Adaptive Multimodal Alignment (UF-AMA). The approach begins with the development of a cross-modal feature fusion network. This network integrates Transformer encoders with multi-head cross-attention modules to facilitate the deep integration of eye-tracking data and EEG signals.

Following this, the framework employs a confidence-aware screening mechanism. This component dynamically evaluates the predictive reliability of each modality branch when processing target domain samples. Based on this assessment, samples are segmented into distinct quality subsets. Subsequently, the system applies global consistency alignment and cross-modal distillation tailored to these subsets.

The final component of UF-AMA is a multi-level domain adaptation framework. This module works to jointly optimize the marginal and conditional distributions of both global fusion features and local modality-specific features. By doing so, it mitigates cross-domain distribution shifts across various granularities. Comprehensive experiments conducted on the SEED and SEED-IV datasets reveal that UF-AMA delivers state-of-the-art (SOTA) results in both cross-session and cross-subject tasks. The source code for this project is accessible at: https://github.com/BetterCoderLab/UF-AMA.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC