Purpose: Gaze-following, the task of inferring where individuals are looking, has been widely studied in computer vision, advancing research in visual attention modeling, social scene understanding, and human-robot interaction. However, gaze-following has never been explored in the operating room (OR), a complex, high-stakes environment where visual attention plays an important role in surgical workflow analysis. In this work, we introduce the concept of gaze-following to the surgical domain, and demonstrate its great potential for understanding clinical roles, surgical phases, and team communications in the OR. Methods: We extend the 4D-OR dataset with gaze-following annotations, and extend the Team-OR dataset with gaze-following and a new team communication activity annotations. Then, we propose novel approaches to address clinical role prediction, surgical phase recognition, and team communication detection using a gaze-following model. For role and phase recognition, we propose a gaze heatmap-based approach that uses gaze predictions solely; for team communication detection, we train a spatial-temporal model in a self-supervised way that encodes gaze-based clip features, and then feed the features into a temporal activity detection model. Results: Experimental results on the 4D-OR and Team-OR datasets demonstrate that our approach achieves state-of-the-art performance on all downstream tasks. Quantitatively, our approach obtains F1 scores of 0.92 for clinical role prediction and 0.95 for surgical phase recognition. Furthermore, it significantly outperforms existing baselines in team communication detection, improving previous best performances by over 30%. Conclusion: We introduce gaze-following in the OR as a novel research direction in surgical data science, highlighting its great potential to advance surgical workflow analysis in computer-assisted interventions.

翻译：目的：视线跟随，即推断个体注视方向的任务，已在计算机视觉领域得到广泛研究，推动了视觉注意力建模、社交场景理解和人机交互等领域的发展。然而，视线跟随在手术室这一复杂的高风险环境中尚未被探索——在此场景中，视觉注意力在手术流程分析中扮演着关键角色。本研究将视线跟随概念引入手术领域，并展示了其在理解手术室中的临床角色、手术阶段及团队沟通方面的巨大潜力。方法：我们扩展了4D-OR数据集，添加了视线跟随标注；同时扩展了Team-OR数据集，增加了视线跟随和新的团队沟通活动标注。随后，我们提出了基于视线跟随模型的新方法，用于临床角色预测、手术阶段识别和团队沟通检测。针对角色与阶段识别，我们提出了一种仅使用视线预测的视线热力图方法；针对团队沟通检测，我们以自监督方式训练了一个时空模型，编码基于视线的片段特征，并将特征输入时序活动检测模型。结果：在4D-OR和Team-OR数据集上的实验结果表明，我们的方法在所有下游任务中均达到最先进性能。量化方面，临床角色预测的F1分数达到0.92，手术阶段识别达到0.95。此外，在团队沟通检测中，我们的方法显著优于现有基线，将先前最佳性能提升了30%以上。结论：我们将手术室中的视线跟随引入手术数据科学的新研究方向，强调了其在计算机辅助干预中推动手术流程分析的巨大潜力。