This paper presents the design and results of the "PEg TRAnsfert Workflow recognition" (PETRAW) challenge whose objective was to develop surgical workflow recognition methods based on one or several modalities, among video, kinematic, and segmentation data, in order to study their added value. The PETRAW challenge provided a data set of 150 peg transfer sequences performed on a virtual simulator. This data set was composed of videos, kinematics, semantic segmentation, and workflow annotations which described the sequences at three different granularity levels: phase, step, and activity. Five tasks were proposed to the participants: three of them were related to the recognition of all granularities with one of the available modalities, while the others addressed the recognition with a combination of modalities. Average application-dependent balanced accuracy (AD-Accuracy) was used as evaluation metric to take unbalanced classes into account and because it is more clinically relevant than a frame-by-frame score. Seven teams participated in at least one task and four of them in all tasks. Best results are obtained with the use of the video and the kinematics data with an AD-Accuracy between 93% and 90% for the four teams who participated in all tasks. The improvement between video/kinematic-based methods and the uni-modality ones was significant for all of the teams. However, the difference in testing execution time between the video/kinematic-based and the kinematic-based methods has to be taken into consideration. Is it relevant to spend 20 to 200 times more computing time for less than 3% of improvement? The PETRAW data set is publicly available at www.synapse.org/PETRAW to encourage further research in surgical workflow recognition.

翻译：本文介绍了“PEg TRAnsfert Workflow 识别”（PETRAW）挑战赛的设计与结果，其目标是基于视频、运动学和分割数据等一种或多种模态开发手术工作流识别方法，以研究这些模态的附加价值。PETRAW 挑战赛提供了一个包含 150 个在虚拟模拟器上执行的夹持转移序列的数据集。该数据集由视频、运动学数据、语义分割和工作流标注组成，这些标注从三个不同粒度层面（阶段、步骤和活动）描述了序列。比赛为参与者设置了五项任务：其中三项任务涉及使用单一可用模态对全部粒度进行识别，其余任务则要求使用多模态组合进行识别。评估指标采用平均依赖应用平衡精度（AD-Accuracy），以处理类别不平衡问题，并因其比逐帧评分更具临床相关性。共有七支队伍参与了至少一项任务，其中四支队伍参与了全部任务。在参与所有任务的四支队伍中，使用视频与运动学数据结合的方法取得了最佳结果，AD-Accuracy 介于 93% 至 90% 之间。对所有队伍而言，基于视频/运动学的方法与单模态方法相比均有显著提升。然而，需考虑基于视频/运动学方法与基于运动学方法在测试执行时间上的差异：为获得不到 3% 的性能提升而花费 20 至 200 倍的计算时间是否值得？PETRAW 数据集已在 www.synapse.org/PETRAW 上公开，以鼓励手术工作流识别领域的进一步研究。