This paper presents a novel visual-language model called DFER-CLIP, which is based on the CLIP model and designed for in-the-wild Dynamic Facial Expression Recognition (DFER). Specifically, the proposed DFER-CLIP consists of a visual part and a textual part. For the visual part, based on the CLIP image encoder, a temporal model consisting of several Transformer encoders is introduced for extracting temporal facial expression features, and the final feature embedding is obtained as a learnable "class" token. For the textual part, we use as inputs textual descriptions of the facial behaviour that is related to the classes (facial expressions) that we are interested in recognising -- those descriptions are generated using large language models, like ChatGPT. This, in contrast to works that use only the class names and more accurately captures the relationship between them. Alongside the textual description, we introduce a learnable token which helps the model learn relevant context information for each expression during training. Extensive experiments demonstrate the effectiveness of the proposed method and show that our DFER-CLIP also achieves state-of-the-art results compared with the current supervised DFER methods on the DFEW, FERV39k, and MAFW benchmarks. Code is publicly available at https://github.com/zengqunzhao/DFER-CLIP.

翻译：本文提出了一种基于CLIP模型的新型视觉语言模型DFER-CLIP，专为自然场景下的动态面部表情识别（Dynamic Facial Expression Recognition, DFER）任务而设计。具体而言，所提出的DFER-CLIP包含视觉部分和文本部分。在视觉部分中，基于CLIP图像编码器，引入了一个由若干Transformer编码器组成的时序模型来提取动态面部表情特征，并以可学习的“类别”令牌作为最终的特征嵌入。在文本部分中，我们使用与待识别类别（面部表情）相关的面部行为文本描述作为输入——这些描述由ChatGPT等大型语言模型生成。与仅使用类别名称的方法不同，这种方式能更精确地捕捉类别间的关联。除了文本描述外，我们还引入了一个可学习的令牌，帮助模型在训练过程中学习每个表情的相关上下文信息。大量实验证明了该方法的有效性，并显示我们的DFER-CLIP在DFEW、FERV39k和MAFW基准数据集上均取得了与当前有监督DFER方法相比的最优结果。代码已开源发布于https://github.com/zengqunzhao/DFER-CLIP。