We developed an American Sign Language (ASL) learning platform in a Virtual Reality (VR) environment to facilitate immersive interaction and real-time feedback for ASL learners. We describe the first game to use an interactive teaching style in which users learn from a fluent signing avatar and the first implementation of ASL sign recognition using deep learning within the VR environment. Advanced motion-capture technology powers an expressive ASL teaching avatar within an immersive three-dimensional environment. The teacher demonstrates an ASL sign for an object, prompting the user to copy the sign. Upon the user's signing, a third-party plugin executes the sign recognition process alongside a deep learning model. Depending on the accuracy of a user's sign production, the avatar repeats the sign or introduces a new one. We gathered a 3D VR ASL dataset from fifteen diverse participants to power the sign recognition model. The proposed deep learning model's training, validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The functional prototype can teach sign language vocabulary and be successfully adapted as an interactive ASL learning platform in VR.

翻译：我们开发了一个基于虚拟现实（VR）环境的美国手语（ASL）学习平台，旨在为ASL学习者提供沉浸式交互与实时反馈。本文描述了首个采用互动教学风格（用户通过流畅演示手语的虚拟化身进行学习）的游戏，也是首个在VR环境中利用深度学习实现ASL手语识别的系统。先进的动作捕捉技术驱动着一个富有表现力的ASL教学虚拟化身，使其在沉浸式三维环境中演示目标物体的手语动作。教师化身示范一个ASL手语后，提示用户模仿该手势。当用户做出手势时，第三方插件协同深度学习模型执行手语识别流程。根据用户手势的准确度，虚拟化身会重复该手势或引入新手势。我们收集了来自15位不同参与者的3D VR ASL数据集以支撑手语识别模型。所提出的深度学习模型在训练集、验证集和测试集上的准确率分别为90.12%、89.37%和86.66%。该功能原型能够教授手语词汇，并可成功转化为VR环境中的交互式ASL学习平台。