In surgical skill assessment, Objective Structured Assessments of Technical Skills (OSATS scores) and the Global Rating Scale (GRS) are established tools for evaluating the performance of surgeons during training. These metrics, coupled with feedback on their performance, enable surgeons to improve and achieve standards of practice. Recent studies on the open-source dataset JIGSAW, which contains both GRS and OSATS labels, have focused on regressing GRS scores from kinematic signals, video data, or a combination of both. In this paper, we argue that regressing the GRS score, a unitless value, by itself is too restrictive, and variations throughout the surgical trial do not hold significant clinical meaning. To address this gap, we developed a recurrent transformer model that outputs the surgeon's performance throughout their training session by relating the model's hidden states to five OSATS scores derived from kinematic signals. These scores are averaged and aggregated to produce a GRS prediction, enabling assessment of the model's performance against the state-of-the-art (SOTA). We report Spearman's Correlation Coefficient (SCC), demonstrating that our model outperforms SOTA models for all tasks, except for Suturing under the leave-one-subject-out (LOSO) scheme (SCC 0.68-0.89), while achieving comparable performance for suturing and across tasks under the leave-one-user-out (LOUO) scheme (SCC 0.45-0.68) and beating SOTA for Needle Passing (0.69). We argue that relating final OSATS scores to short instances throughout a surgeon's procedure is more clinically meaningful than a single GRS score. This approach also allows us to translate quantitative predictions into qualitative feedback, which is crucial for any automated surgical skill assessment pipeline. A senior surgeon validated our model's behaviour and agreed with the semi-supervised predictions 77 \% (p = 0.006) of the time.

翻译：在外科手术技能评估中，客观结构化技术技能评估（OSATS评分）和整体评分量表（GRS）是用于评估外科医生培训期间表现的成熟工具。这些指标结合对其表现的反馈，使外科医生能够改进并达到实践标准。近期针对包含GRS和OSATS标签的开源数据集JIGSAW的研究，主要集中在从运动学信号、视频数据或两者结合中回归GRS分数。本文认为，单独回归GRS分数（一个无量纲值）过于局限，且手术试验过程中的变化不具有显著的临床意义。为弥补这一不足，我们开发了一种循环Transformer模型，通过将模型隐藏状态与从运动学信号推导出的五个OSATS分数相关联，输出外科医生在整个训练过程中的表现。这些分数经过平均和聚合以生成GRS预测，从而能够将模型性能与最先进（SOTA）方法进行比较评估。我们报告了斯皮尔曼相关系数（SCC），结果表明除在留一受试者（LOSO）方案下的缝合任务（SCC 0.68-0.89）外，我们的模型在所有任务上均优于SOTA模型；同时在留一用户（LOUO）方案下的缝合任务及跨任务中达到可比性能（SCC 0.45-0.68），并在穿针任务上超越SOTA（0.69）。我们认为，将最终OSATS分数与外科医生手术过程中的短期实例相关联，比单一GRS分数更具临床意义。这种方法还使我们能够将定量预测转化为定性反馈，这对任何自动化手术技能评估流程都至关重要。一位资深外科医生验证了我们模型的行为，并在77%的情况下（p = 0.006）认同半监督预测结果。