In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on EndoVis2017 and EndoVis2018 datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery.

翻译：本文提出了一种新颖的文本提示式手术器械分割方法，旨在克服微创手术中手术器械多样性与差异性带来的挑战。我们将任务重新定义为文本可提示模式，从而实现对手术器械更细致的理解，并增强对新器械类型的适应性。受视觉-语言模型最新进展的启发，我们利用预训练的图像和文本编码器作为模型骨干架构，并设计了一个包含基于注意力与基于卷积的提示机制的文本提示式掩码解码器，用于预测手术器械分割结果。通过一种新的提示混合机制，我们为每种手术器械引入多个文本提示，显著提升了分割性能。此外，我们还引入了硬器械区域强化模块，以改善图像特征理解与分割精度。在EndoVis2017与EndoVis2018数据集上的大量实验表明，该模型具有优越的性能与良好的泛化能力。据我们所知，这是首次将可提示方法应用于手术器械分割领域，为机器人辅助手术的实际应用提供了重要潜力。