Robust cross-subject emotion recognition from multimodal physiological signals remains a challenging problem, primarily due to modality heterogeneity and inter-subject distribution shift. To tackle these challenges, we propose a novel adaptive learning framework named Hierarchical Attention and Dynamic Uniform Alignment (HADUA). Our approach unifies the learning of multimodal representations with domain adaptation. First, we design a hierarchical attention module that explicitly models intra-modal temporal dynamics and inter-modal semantic interactions (e.g., between electroencephalogram(EEG) and eye movement(EM)), yielding discriminative and semantically coherent fused features. Second, to overcome the noise inherent in pseudo-labels during adaptation, we introduce a confidence-aware Gaussian weighting scheme that smooths the supervision from target-domain samples by down-weighting uncertain instances. Third, a uniform alignment loss is employed to regularize the distribution of pseudo-labels across classes, thereby mitigating imbalance and stabilizing conditional distribution matching. Extensive experiments on multiple cross-subject emotion recognition benchmarks show that HADUA consistently surpasses existing state-of-the-art methods in both accuracy and robustness, validating its effectiveness in handling modality gaps, noisy pseudo-labels, and class imbalance. Taken together, these contributions offer a practical and generalizable solution for building robust cross-subject affective computing systems.

翻译：基于多模态生理信号的鲁棒跨被试情绪识别仍是一个具有挑战性的问题，主要源于模态异质性与被试间分布偏移。为应对这些挑战，我们提出了一种名为分层注意力与动态均匀对齐（HADUA）的新型自适应学习框架。该方法将多模态表征学习与域适应进行统一。首先，我们设计了一个分层注意力模块，显式建模模态内时序动态与模态间语义交互（例如脑电图与眼动信号之间），从而生成判别性强且语义连贯的融合特征。其次，为克服适应过程中伪标签固有的噪声，我们引入了一种置信感知的高斯加权方案，通过对不确定实例进行降权来平滑目标域样本的监督信号。第三，采用均匀对齐损失来规范化各类别间伪标签的分布，从而缓解不平衡问题并稳定条件分布匹配。在多个跨被试情绪识别基准上的大量实验表明，HADUA在准确性与鲁棒性上均持续超越现有最先进方法，验证了其在处理模态差异、噪声伪标签和类别不平衡问题上的有效性。综上所述，这些贡献为构建鲁棒的跨被试情感计算系统提供了一个实用且可推广的解决方案。