In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.

翻译：本文提出了一种新颖的文本可提示手术器械分割方法，以克服微创手术中手术器械多样性与差异性带来的挑战。我们将该任务重新定义为文本可提示模式，从而实现对手术器械更精细的理解，并增强对新器械类型的适应性。受视觉-语言模型最新进展的启发，我们采用预训练的图像和文本编码器作为模型主干，并设计了一个基于注意力机制与卷积提示方案的文本可提示掩码解码器，用于手术器械分割预测。通过新的提示混合机制，模型为每种手术器械利用多个文本提示，从而提升分割性能。此外，我们引入器械难例区域增强模块，以改进图像特征理解能力并提高分割精度。在多个手术器械分割数据集上的大量实验表明，本模型具有优越的性能及良好的泛化能力。据我们所知，这是首次将可提示方法应用于手术器械分割，为机器人辅助手术领域的实际应用提供了重要潜力。相关代码已开源：https://github.com/franciszzj/TP-SIS。