Robot-assisted surgery has made significant progress, with instrument segmentation being a critical factor in surgical intervention quality. It serves as the building block to facilitate surgical robot navigation and surgical education for the next generation of operating intelligence. Although existing methods have achieved accurate instrument segmentation results, they simultaneously generate segmentation masks for all instruments, without the capability to specify a target object and allow an interactive experience. This work explores a new task of Referring Surgical Video Instrument Segmentation (RSVIS), which aims to automatically identify and segment the corresponding surgical instruments based on the given language expression. To achieve this, we devise a novel Video-Instrument Synergistic Network (VIS-Net) to learn both video-level and instrument-level knowledge to boost performance, while previous work only used video-level information. Meanwhile, we design a Graph-based Relation-aware Module (GRM) to model the correlation between multi-modal information (i.e., textual description and video frame) to facilitate the extraction of instrument-level information. We are also the first to produce two RSVIS datasets to promote related research. Our method is verified on these datasets, and experimental results exhibit that the VIS-Net can significantly outperform existing state-of-the-art referring segmentation methods. Our code and our datasets will be released upon the publication of this work.

翻译：机器人辅助手术已取得显著进展，其中器械分割是影响手术干预质量的关键因素。作为推动手术机器人导航和下一代手术智能教学的基础模块，现有方法虽能实现精确的器械分割结果，却无法指定目标对象并提供交互式体验——它们仅能同时生成所有器械的分割掩码。本文探索了指代性手术视频器械分割这一新任务，旨在根据给定的语言表达自动识别并分割对应手术器械。为实现该目标，我们设计了新型视频-器械协同网络，通过同时学习视频级和器械级知识来提升性能，而先前工作仅利用视频级信息。同时，我们提出基于图的关系感知模块来建模多模态信息（即文本描述与视频帧）之间的关联，以促进器械级信息的提取。作为首个工作，我们还构建了两个RSVIS数据集以推动相关研究。方法在这些数据集上得到验证，实验结果表明VIS-Net能够显著超越现有最先进的指代性分割方法。本文代码与数据集将在论文发表后开源。