Video-based facial affect analysis has recently attracted increasing attention owing to its critical role in human-computer interaction. Previous studies mainly focus on developing various deep learning architectures and training them in a fully supervised manner. Although significant progress has been achieved by these supervised methods, the longstanding lack of large-scale high-quality labeled data severely hinders their further improvements. Motivated by the recent success of self-supervised learning in computer vision, this paper introduces a self-supervised approach, termed Self-supervised Video Facial Affect Perceiver (SVFAP), to address the dilemma faced by supervised methods. Specifically, SVFAP leverages masked facial video autoencoding to perform self-supervised pre-training on massive unlabeled facial videos. Considering that large spatiotemporal redundancy exists in facial videos, we propose a novel temporal pyramid and spatial bottleneck Transformer as the encoder of SVFAP, which not only largely reduces computational costs but also achieves excellent performance. To verify the effectiveness of our method, we conduct experiments on nine datasets spanning three downstream tasks, including dynamic facial expression recognition, dimensional emotion recognition, and personality recognition. Comprehensive results demonstrate that SVFAP can learn powerful affect-related representations via large-scale self-supervised pre-training and it significantly outperforms previous state-of-the-art methods on all datasets. Code is available at https://github.com/sunlicai/SVFAP.

翻译：基于视频的面部情感分析因其在人机交互中的关键作用，近期受到越来越多的关注。先前的研究主要集中于开发各种深度学习架构，并以全监督方式进行训练。尽管这些监督方法已取得显著进展，但长期以来大规模高质量标注数据的缺乏严重阻碍了其进一步改进。受计算机视觉中自监督学习近期成功的启发，本文提出一种自监督方法，称为自监督视频面部情感感知器（SVFAP），以应对监督方法所面临的困境。具体而言，SVFAP利用掩码面部视频自编码技术，在海量无标注面部视频上进行自监督预训练。考虑到面部视频中存在较大的时空冗余，我们提出了一种新颖的时序金字塔与空间瓶颈Transformer作为SVFAP的编码器，该设计不仅大幅降低了计算成本，同时取得了优异的性能。为验证方法的有效性，我们在涵盖三个下游任务的九个数据集上进行了实验，包括动态面部表情识别、维度情感识别和人格识别。综合结果表明，SVFAP能够通过大规模自监督预训练学习到强大的情感相关表征，并在所有数据集上显著优于先前的最先进方法。代码发布于 https://github.com/sunlicai/SVFAP。