With the strong robusticity on illumination variations, near-infrared (NIR) can be an effective and essential complement to visible (VIS) facial expression recognition in low lighting or complete darkness conditions. However, facial expression recognition (FER) from NIR images presents more challenging problem than traditional FER due to the limitations imposed by the data scale and the difficulty of extracting discriminative features from incomplete visible lighting contents. In this paper, we give the first attempt to deep NIR facial expression recognition and proposed a novel method called near-infrared facial expression transformer (NFER-Former). Specifically, to make full use of the abundant label information in the field of VIS, we introduce a Self-Attention Orthogonal Decomposition mechanism that disentangles the expression information and spectrum information from the input image, so that the expression features can be extracted without the interference of spectrum variation. We also propose a Hypergraph-Guided Feature Embedding method that models some key facial behaviors and learns the structure of the complex correlations between them, thereby alleviating the interference of inter-class similarity. Additionally, we have constructed a large NIR-VIS Facial Expression dataset that includes 360 subjects to better validate the efficiency of NFER-Former. Extensive experiments and ablation studies show that NFER-Former significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

翻译：鉴于近红外（NIR）在光照变化下的强鲁棒性，其在低光照或完全黑暗条件下可作为可见光（VIS）面部表情识别的有效且必要补充。然而，由于数据规模的限制以及从不完整可见光内容中提取判别性特征的困难，基于NIR图像的面部表情识别（FER）比传统FER更具挑战性。本文首次尝试深度NIR面部表情识别，并提出了一种名为近红外面部表情变换器（NFER-Former）的新方法。具体而言，为充分利用VIS领域丰富的标签信息，我们引入了一种自注意力正交分解机制，从输入图像中解耦表情信息与频谱信息，使表情特征能够不受频谱变化干扰而提取。我们还提出了一种超图引导的特征嵌入方法，该方法对关键面部行为进行建模，并学习它们之间复杂关联的结构，从而缓解类间相似性的干扰。此外，我们构建了一个包含360名受试者的大型NIR-VIS面部表情数据集，以更好地验证NFER-Former的有效性。大量实验和消融研究表明，NFER-Former显著提升了NIR FER的性能，并在仅有的两个可用NIR FER数据集（Oulu-CASIA和Large-HFE）上取得了最先进的结果。