Foundation Vision-Language Models (VLMs) trained using large-scale open-domain images and text pairs have recently been adapted to develop Vision-Language Segmentation Models (VLSMs) that allow providing text prompts during inference to guide image segmentation. If robust and powerful VLSMs can be built for medical images, it could aid medical professionals in many clinical tasks where they must spend substantial time delineating the target structure of interest. VLSMs for medical images resort to fine-tuning base VLM or VLSM pretrained on open-domain natural image datasets due to fewer annotated medical image datasets; this fine-tuning is resource-consuming and expensive as it usually requires updating all or a significant fraction of the pretrained parameters. Recently, lightweight blocks called adapters have been proposed in VLMs that keep the pretrained model frozen and only train adapters during fine-tuning, substantially reducing the computing resources required. We introduce a novel adapter, VLSM-Adapter, that can fine-tune pretrained vision-language segmentation models using transformer encoders. Our experiments in widely used CLIP-based segmentation models show that with only 3 million trainable parameters, the VLSM-Adapter outperforms state-of-the-art and is comparable to the upper bound end-to-end fine-tuning. The source code is available at: https://github.com/naamiinepal/vlsm-adapter.

翻译：基于大规模开放域图像-文本对训练的基础视觉语言模型（VLMs）近期已被用于开发视觉语言分割模型（VLSMs），该类模型允许在推理过程中提供文本提示以引导图像分割。若能为医学图像构建稳健且强大的VLSMs，将有助于医疗专业人员在许多需耗费大量时间勾画目标结构的临床任务中提升效率。由于标注医学图像数据集相对稀缺，面向医学图像的VLSMs通常通过对在开放域自然图像数据集上预训练的基础VLM或VLSM进行微调实现；这种微调过程通常需要更新全部或大部分预训练参数，导致计算资源消耗巨大且成本高昂。近期，VLMs领域提出了称为适配器的轻量级模块，该模块可在微调期间保持预训练模型冻结状态，仅训练适配器参数，从而显著降低所需计算资源。本文提出一种新型适配器——VLSM-Adapter，能够利用Transformer编码器对预训练的视觉语言分割模型进行微调。我们在广泛使用的基于CLIP的分割模型上开展实验，结果表明仅使用300万个可训练参数，VLSM-Adapter即能超越现有最优方法，并与端到端微调的性能上限相当。源代码已发布于：https://github.com/naamiinepal/vlsm-adapter。