Video-based facial affect analysis has recently attracted increasing attention owing to its critical role in human-computer interaction. Previous studies mainly focus on developing various deep learning architectures and training them in a fully supervised manner. Although significant progress has been achieved by these supervised methods, the longstanding lack of large-scale high-quality labeled data severely hinders their further improvements. Motivated by the recent success of self-supervised learning in computer vision, this paper introduces a self-supervised approach, termed Self-supervised Video Facial Affect Perceiver (SVFAP), to address the dilemma faced by supervised methods. Specifically, SVFAP leverages masked facial video autoencoding to perform self-supervised pre-training on massive unlabeled facial videos. Considering that large spatiotemporal redundancy exists in facial videos, we propose a novel temporal pyramid and spatial bottleneck Transformer as the encoder of SVFAP, which not only enjoys low computational cost but also achieves excellent performance. To verify the effectiveness of our method, we conduct experiments on nine datasets spanning three downstream tasks, including dynamic facial expression recognition, dimensional emotion recognition, and personality recognition. Comprehensive results demonstrate that SVFAP can learn powerful affect-related representations via large-scale self-supervised pre-training and it significantly outperforms previous state-of-the-art methods on all datasets. Codes will be available at https://github.com/sunlicai/SVFAP.

翻译：基于视频的面部情感分析因其在人机交互中的关键作用，近年来受到日益广泛的关注。以往研究主要集中于开发多种深度学习架构，并以全监督方式进行训练。尽管这些监督方法取得了显著进展，但长期缺乏大规模高质量标注数据严重阻碍了其进一步发展。受自监督学习在计算机视觉领域近期成功的启发，本文提出了一种名为"自监督视频面部情感感知器"（SVFAP）的自监督方法，以解决监督方法面临的困境。具体而言，SVFAP利用掩码面部视频自编码技术，在大量无标注面部视频上进行自监督预训练。考虑到面部视频中存在显著的空时冗余性，我们提出了一种新颖的时间金字塔与空间瓶颈Transformer作为SVFAP的编码器，该编码器不仅计算成本低，而且性能优异。为验证方法的有效性，我们在涵盖三个下游任务（包括动态面部表情识别、维度情感识别和人格识别）的九个数据集上进行了实验。综合结果表明，SVFAP能够通过大规模自监督预训练学习到强大的情感相关表征，并在所有数据集上显著优于先前的最优方法。代码将在https://github.com/sunlicai/SVFAP 提供。