Sign language serves as a non-vocal means of communication, transmitting information and significance through gestures, facial expressions, and bodily movements. The majority of current approaches for sign language recognition (SLR) and translation rely on RGB video inputs, which are vulnerable to fluctuations in the background. Employing a keypoint-based strategy not only mitigates the effects of background alterations but also substantially diminishes the computational demands of the model. Nevertheless, contemporary keypoint-based methodologies fail to fully harness the implicit knowledge embedded in keypoint sequences. To tackle this challenge, our inspiration is derived from the human cognition mechanism, which discerns sign language by analyzing the interplay between gesture configurations and supplementary elements. We propose a multi-stream keypoint attention network to depict a sequence of keypoints produced by a readily available keypoint estimator. In order to facilitate interaction across multiple streams, we investigate diverse methodologies such as keypoint fusion strategies, head fusion, and self-distillation. The resulting framework is denoted as MSKA-SLR, which is expanded into a sign language translation (SLT) model through the straightforward addition of an extra translation network. We carry out comprehensive experiments on well-known benchmarks like Phoenix-2014, Phoenix-2014T, and CSL-Daily to showcase the efficacy of our methodology. Notably, we have attained a novel state-of-the-art performance in the sign language translation task of Phoenix-2014T. The code and models can be accessed at: https://github.com/sutwangyan/MSKA.

翻译：手语作为一种非言语交流方式，通过手势、面部表情和身体动作传递信息与意义。当前大多数手语识别（SLR）和翻译方法依赖RGB视频输入，易受背景波动影响。采用基于关键点的策略不仅能减轻背景变化的影响，还能显著降低模型的计算需求。然而，现有的基于关键点的方法未能充分利用关键点序列中蕴含的隐式知识。为解决这一挑战，我们从人类认知机制中汲取灵感，该机制通过分析手语配置与辅助元素之间的相互作用来辨识手语。本文提出一种多流关键点注意力网络，用于描述由现有关键点估计器生成的关键点序列。为促进多流间的交互，我们探索了多种方法，包括关键点融合策略、头部融合和自蒸馏。所得框架记为MSKA-SLR，通过简单添加额外的翻译网络即可扩展为手语翻译（SLT）模型。我们在Phoenix-2014、Phoenix-2014T和CSL-Daily等知名基准上进行了全面实验，以展示我们方法的有效性。值得注意的是，我们在Phoenix-2014T的手语翻译任务中取得了新的最佳性能。代码和模型可在https://github.com/sutwangyan/MSKA获取。