Sign Language Recognition (SLR) is an essential yet challenging task since sign language is performed with the fast and complex movement of hand gestures, body posture, and even facial expressions. %Skeleton Aware Multi-modal Sign Language Recognition In this work, we focused on investigating two questions: how fine-tuning on datasets from other sign languages helps improve sign recognition quality, and whether sign recognition is possible in real-time without using GPU. Three different languages datasets (American sign language WLASL, Turkish - AUTSL, Russian - RSL) have been used to validate the models. The average speed of this system has reached 3 predictions per second, which meets the requirements for the real-time scenario. This model (prototype) will benefit speech or hearing impaired people talk with other trough internet. We also investigated how the additional training of the model in another sign language affects the quality of recognition. The results show that further training of the model on the data of another sign language almost always leads to an improvement in the quality of gesture recognition. We also provide code for reproducing model training experiments, converting models to ONNX format, and inference for real-time gesture recognition.

翻译：手语识别（SLR）是一项重要但具有挑战性的任务，因为手语通过快速且复杂的手势、身体姿态甚至面部表情来表现。%骨架感知多模态手语识别 在这项工作中，我们重点研究了两个问题：对来自其他手语数据集进行微调如何有助于提高手语识别质量，以及在不使用GPU的情况下能否实现实时手语识别。我们使用了三种不同语言的数据集（美国手语WLASL、土耳其语AUTSL、俄语RSL）来验证模型。该系统的平均速度达到每秒3次预测，满足实时场景的要求。该模型（原型）将有助于听障或语障人士通过互联网与他人交流。我们还研究了在另一种手语上对模型进行额外训练对识别质量的影响。结果表明，在另一种手语数据上进一步训练模型几乎总能提高手势识别质量。我们还提供了用于复现模型训练实验、将模型转换为ONNX格式以及进行实时手势识别推理的代码。