Semi-supervised learning has emerged as a promising approach to tackle the challenge of label scarcity in facial expression recognition (FER) task. However, current state-of-the-art methods primarily focus on one side of the coin, i.e., generating high-quality pseudo-labels, while overlooking the other side: enhancing expression-relevant representations. In this paper, we unveil both sides of the coin by proposing a unified framework termed hierarchicaL dEcoupling And Fusing (LEAF) to coordinate expression-relevant representations and pseudo-labels for semi-supervised FER. LEAF introduces a hierarchical expression-aware aggregation strategy that operates at three levels: semantic, instance, and category. (1) At the semantic and instance levels, LEAF decouples representations into expression-agnostic and expression-relevant components, and adaptively fuses them using learnable gating weights. (2) At the category level, LEAF assigns ambiguous pseudo-labels by decoupling predictions into positive and negative parts, and employs a consistency loss to ensure agreement between two augmented views of the same image. Extensive experiments on benchmark datasets demonstrate that by unveiling and harmonizing both sides of the coin, LEAF outperforms state-of-the-art semi-supervised FER methods, effectively leveraging both labeled and unlabeled data. Moreover, the proposed expression-aware aggregation strategy can be seamlessly integrated into existing semi-supervised frameworks, leading to significant performance gains.

翻译：半监督学习已成为应对面部表情识别（FER）任务中标签稀缺挑战的一种有前途的方法。然而，当前最先进的方法主要关注硬币的一面，即生成高质量的伪标签，而忽视了另一面：增强与表情相关的表征。在本文中，我们通过提出一个名为分层解耦与融合（LEAF）的统一框架，揭示了硬币的两面，旨在协调半监督FER中的表情相关表征和伪标签。LEAF引入了一种分层表情感知聚合策略，该策略在三个层级上运行：语义层、实例层和类别层。（1）在语义和实例层级，LEAF将表征解耦为表情无关和表情相关成分，并通过可学习的门控权重自适应地融合它们。（2）在类别层级，LEAF通过将预测解耦为正负两部分来分配模糊伪标签，并采用一致性损失确保同一图像的两个增强视图之间的一致性。在基准数据集上的大量实验表明，通过揭示并协调硬币的两面，LEAF优于最先进的半监督FER方法，有效利用了标注和未标注数据。此外，所提出的表情感知聚合策略可以无缝集成到现有的半监督框架中，从而带来显著的性能提升。