Purpose: The recent Segment Anything Model (SAM) has demonstrated impressive performance with point, text or bounding box prompts, in various applications. However, in safety-critical surgical tasks, prompting is not possible due to (i) the lack of per-frame prompts for supervised learning, (ii) it is unrealistic to prompt frame-by-frame in a real-time tracking application, and (iii) it is expensive to annotate prompts for offline applications. Methods: We develop Surgical-DeSAM to generate automatic bounding box prompts for decoupling SAM to obtain instrument segmentation in real-time robotic surgery. We utilise a commonly used detection architecture, DETR, and fine-tuned it to obtain bounding box prompt for the instruments. We then empolyed decoupling SAM (DeSAM) by replacing the image encoder with DETR encoder and fine-tune prompt encoder and mask decoder to obtain instance segmentation for the surgical instruments. To improve detection performance, we adopted the Swin-transformer to better feature representation. Results: The proposed method has been validated on two publicly available datasets from the MICCAI surgical instruments segmentation challenge EndoVis 2017 and 2018. The performance of our method is also compared with SOTA instrument segmentation methods and demonstrated significant improvements with dice metrics of 89.62 and 90.70 for the EndoVis 2017 and 2018. Conclusion: Our extensive experiments and validations demonstrate that Surgical-DeSAM enables real-time instrument segmentation without any additional prompting and outperforms other SOTA segmentation methods.

翻译：目的：近年提出的Segment Anything Model（SAM）凭借点、文本或边界框提示，在各类应用中展现出卓越性能。然而，在安全关键型手术任务中，提示机制难以应用，原因在于：（i）监督学习缺乏逐帧提示；（ii）实时跟踪应用中逐帧提示不切实际；（iii）离线应用场景的提示标注成本高昂。方法：我们提出Surgical-DeSAM方法，通过生成自动边界框提示来解耦SAM，以实现机器人手术中的实时器械分割。该方法采用通用检测架构DETR，经微调后生成器械的边界框提示；随后通过将图像编码器替换为DETR编码器，并微调提示编码器与掩码解码器，构建解耦式SAM（DeSAM），最终获得手术器械的实例分割结果。为提升检测性能，我们采用Swin-Transformer优化特征表示。结果：所提方法已在MICCAI手术器械分割挑战赛EndoVis 2017和2018的两个公开数据集上完成验证。与当前最优（SOTA）器械分割方法相比，本方法性能显著提升，在EndoVis 2017与2018数据集上的Dice指标分别达到89.62和90.70。结论：大量实验与验证表明，Surgical-DeSAM无需额外提示即可实现实时器械分割，性能优于其他SOTA分割方法。