Sign languages are visual languages which convey information by signers' handshape, facial expression, body movement, and so forth. Due to the inherent restriction of combinations of these visual ingredients, there exist a significant number of visually indistinguishable signs (VISigns) in sign languages, which limits the recognition capacity of vision neural networks. To mitigate the problem, we propose the Natural Language-Assisted Sign Language Recognition (NLA-SLR) framework, which exploits semantic information contained in glosses (sign labels). First, for VISigns with similar semantic meanings, we propose language-aware label smoothing by generating soft labels for each training sign whose smoothing weights are computed from the normalized semantic similarities among the glosses to ease training. Second, for VISigns with distinct semantic meanings, we present an inter-modality mixup technique which blends vision and gloss features to further maximize the separability of different signs under the supervision of blended labels. Besides, we also introduce a novel backbone, video-keypoint network, which not only models both RGB videos and human body keypoints but also derives knowledge from sign videos of different temporal receptive fields. Empirically, our method achieves state-of-the-art performance on three widely-adopted benchmarks: MSASL, WLASL, and NMFs-CSL. Codes are available at https://github.com/FangyunWei/SLRT.

翻译：手语是一种视觉语言，通过手形、面部表情、身体动作等视觉成分传递信息。由于这些视觉成分组合的内在限制，手语中存在大量视觉上难以区分的符号（VISigns），这限制了视觉神经网络的识别能力。为缓解这一问题，我们提出了自然语言辅助的手语识别（NLA-SLR）框架，该框架利用手势标签（glosses）中包含的语义信息。首先，针对语义相似的手语符号，我们提出语言感知的标签平滑技术，通过为每个训练样本生成软标签（平滑权重基于各手势标签之间的归一化语义相似度计算）来简化训练过程。其次，针对语义不同的手语符号，我们提出跨模态混合技术，将视觉特征与手势标签特征融合，在混合标签的监督下进一步最大化不同符号的可区分性。此外，我们还引入了一种新型骨干网络——视频关键点网络，该网络不仅能够处理RGB视频和人体关键点，还能从不同时域感受野的手语视频中提取知识。实验表明，我们的方法在MSASL、WLASL和NMFs-CSL三个广泛采用的基准数据集上达到了最先进的性能。代码已开源：https://github.com/FangyunWei/SLRT。