Automated feedback systems have the potential to provide objective skill assessment for training and evaluation in robot-assisted surgery. In this study, we examine methods to achieve real-time prediction of surgical skill level in real-time based on Objective Structured Assessment of Technical Skills (OSATS) scores. Using data acquired from the da Vinci Surgical System, we carry out three main analyses, focusing on model design, their real-time performance, and their skill-level-based cross-validation training. For the model design, we evaluate the effectiveness of multimodal deep learning models for predicting surgical skill levels using synchronized kinematic and vision data. Our models include separate unimodal baselines and fusion architectures that integrate features from both modalities and are evaluated using mean Spearman's correlation coefficients, demonstrating that the fusion model consistently outperforms unimodal models for real-time predictions. For the real-time performance, we observe the prediction's trend over time and highlight correlation with the surgeon's gestures. For the skill-level-based cross-validation, we separately trained models on surgeons with different skill levels, which showed that high-skill demonstrations allow for better performance than those trained on low-skilled ones and generalize well to similarly skilled participants. Our findings show that multimodal learning allows more stable fine-grained evaluation of surgical performance and highlights the value of expert-level training data for model generalization.

翻译：自动化反馈系统有望为机器人辅助手术的培训与评估提供客观的技能评价。本研究基于客观结构化技术技能评估（OSATS）评分，探讨实现手术技能水平实时预测的方法。利用达芬奇手术系统采集的数据，我们开展了三项核心分析，聚焦于模型设计、其实时性能以及基于技能水平的交叉验证训练。在模型设计方面，我们评估了多模态深度学习模型利用同步运动学与视觉数据预测手术技能水平的有效性。所构建模型包含独立的单模态基线模型与融合两种模态特征的多模态融合架构，通过平均斯皮尔曼相关系数进行评估。结果表明，融合模型在实时预测中持续优于单模态模型。在实时性能分析中，我们观察了预测结果随时间的变化趋势，并揭示了其与外科医生手势动作的相关性。在基于技能水平的交叉验证中，我们分别针对不同技能水平的外科医生训练模型，结果显示基于高水平医生数据训练的模型性能优于基于低水平数据训练的模型，并能良好泛化至同等技能水平的参与者。我们的研究结果表明，多模态学习能够实现更稳定的手术表现细粒度评估，并凸显了专家级训练数据对于模型泛化能力的重要价值。