With wearing masks becoming a new cultural norm, facial expression recognition (FER) while taking masks into account has become a significant challenge. In this paper, we propose a unified multi-branch vision transformer for facial expression recognition and mask wearing classification tasks. Our approach extracts shared features for both tasks using a dual-branch architecture that obtains multi-scale feature representations. Furthermore, we propose a cross-task fusion phase that processes tokens for each task with separate branches, while exchanging information using a cross attention module. Our proposed framework reduces the overall complexity compared with using separate networks for both tasks by the simple yet effective cross-task fusion phase. Extensive experiments demonstrate that our proposed model performs better than or on par with different state-of-the-art methods on both facial expression recognition and facial mask wearing classification task.

翻译：随着佩戴口罩成为新的文化常态，兼顾口罩因素的面部表情识别（FER）已成为一项重大挑战。本文提出一种统一的多分支视觉Transformer，用于面部表情识别与口罩佩戴分类任务。该方法采用双分支架构获取多尺度特征表示，为两项任务提取共享特征。此外，我们提出跨任务融合阶段：通过独立分支处理各任务的令牌，并借助交叉注意力模块交换信息。与为两项任务分别使用独立网络相比，本框架通过简洁高效的跨任务融合阶段降低了整体复杂度。大量实验表明，本模型在面部表情识别与口罩佩戴分类任务上，均优于或持平于多种最先进方法。