Dynamic facial expression recognition (FER) databases provide important data support for affective computing and applications. However, most FER databases are annotated with several basic mutually exclusive emotional categories and contain only one modality, e.g., videos. The monotonous labels and modality cannot accurately imitate human emotions and fulfill applications in the real world. In this paper, we propose MAFW, a large-scale multi-modal compound affective database with 10,045 video-audio clips in the wild. Each clip is annotated with a compound emotional category and a couple of sentences that describe the subjects' affective behaviors in the clip. For the compound emotion annotation, each clip is categorized into one or more of the 11 widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment. To ensure high quality of the labels, we filter out the unreliable annotations by an Expectation Maximization (EM) algorithm, and then obtain 11 single-label emotion categories and 32 multi-label emotion categories. To the best of our knowledge, MAFW is the first in-the-wild multi-modal database annotated with compound emotion annotations and emotion-related captions. Additionally, we also propose a novel Transformer-based expression snippet feature learning method to recognize the compound emotions leveraging the expression-change relations among different emotions and modalities. Extensive experiments on MAFW database show the advantages of the proposed method over other state-of-the-art methods for both uni- and multi-modal FER. Our MAFW database is publicly available from https://mafw-database.github.io/MAFW.

翻译：动态面部表情识别（FER）数据库为情感计算及其应用提供了重要的数据支撑。然而，多数FER数据库仅标注几种互斥的基本情感类别，且仅包含单一模态（如视频）。单调的标签与模态无法准确模拟人类情感，也难以满足真实世界应用需求。本文提出MAFW——一个包含10,045个野外视频-音频片段的大规模多模态复合情感数据库。每个片段均标注了复合情感类别及若干描述片中主体情感行为的句子。在复合情感标注方面，每个片段被归入11种常用情感中的一种或多种，即愤怒、厌恶、恐惧、快乐、中性、悲伤、惊讶、轻蔑、焦虑、无助与失望。为确保标签质量，我们通过期望最大化（EM）算法滤除不可靠标注，最终获得11个单标签情感类别与32个多标签情感类别。据我们所知，MAFW是首个具备复合情感标注及情感相关文本描述的野外多模态数据库。此外，我们还提出一种基于Transformer的新型表达片段特征学习方法，通过挖掘不同情感与模态间的表达变化关系来识别复合情感。在MAFW数据库上的大量实验表明，所提方法在单模态与多模态FER任务中均优于现有最优方法。我们的MAFW数据库现已公开，访问地址为https://mafw-database.github.io/MAFW。