This paper introduces Video2MR, a mixed reality system that automatically generates 3D sports and exercise instructions from 2D videos. Mixed reality instructions have great potential for physical training, but existing works require substantial time and cost to create these 3D experiences. Video2MR overcomes this limitation by transforming arbitrary instructional videos available online into MR 3D avatars with AI-enabled motion capture (DeepMotion). Then, it automatically enhances the avatar motion through the following augmentation techniques: 1) contrasting and highlighting differences between the user and avatar postures, 2) visualizing key trajectories and movements of specific body parts, 3) manipulation of time and speed using body motion, and 4) spatially repositioning avatars for different perspectives. Developed on Hololens 2 and Azure Kinect, we showcase various use cases, including yoga, dancing, soccer, tennis, and other physical exercises. The study results confirm that Video2MR provides more engaging and playful learning experiences, compared to existing 2D video instructions.

翻译：本文介绍Video2MR，一种从二维视频自动生成三维运动与锻炼教学指令的混合现实系统。混合现实教学在体能训练领域具有巨大潜力，但现有方法创建此类三维体验需要大量时间和成本。Video2MR通过将任意在线教学视频转化为搭载AI动作捕捉技术（DeepMotion）的MR三维虚拟化身，克服了这一限制。系统通过以下增强技术自动优化虚拟化身的运动表现：1）对比并高亮用户与虚拟化身姿态差异，2）可视化特定身体部位的关键轨迹与运动路径，3）利用身体运动实现时间与速度的操控，4）空间重定位虚拟化身以提供多视角观察。基于Hololens 2与Azure Kinect开发的系统展示了涵盖瑜伽、舞蹈、足球、网球及其他体能训练的多类应用场景。研究结果证实，与现有二维视频教学相比，Video2MR能提供更具沉浸感与趣味性的学习体验。