In recent years, various large foundation models have been proposed for image segmentation. There models are often trained on large amounts of data corresponding to general computer vision tasks. Hence, these models do not perform well on medical data. There have been some attempts in the literature to perform parameter-efficient finetuning of such foundation models for medical image segmentation. However, these approaches assume that all the parameters of the model are available for adaptation. But, in many cases, these models are released as APIs or blackboxes, with no or limited access to the model parameters and data. In addition, finetuning methods also require a significant amount of compute, which may not be available for the downstream task. At the same time, medical data can't be shared with third-party agents for finetuning due to privacy reasons. To tackle these challenges, we pioneer a blackbox adaptation technique for prompted medical image segmentation, called BAPS. BAPS has two components - (i) An Image-Prompt decoder (IP decoder) module that generates visual prompts given an image and a prompt, and (ii) A Zero Order Optimization (ZOO) Method, called SPSA-GC that is used to update the IP decoder without the need for backpropagating through the foundation model. Thus, our method does not require any knowledge about the foundation model's weights or gradients. We test BAPS on four different modalities and show that our method can improve the original model's performance by around 4%.

翻译：近年来，针对图像分割任务提出了多种大型基础模型。这些模型通常在大量通用计算机视觉任务数据上训练，因此其在医学数据上的表现并不理想。现有文献中已有一些尝试对这类基础模型进行参数高效微调以实现医学图像分割，但这些方法都假设模型的所有参数均可用于自适应。然而，在许多场景下，这些模型以API或黑盒形式发布，用户无法或仅能有限访问模型参数及数据。此外，微调方法还需大量计算资源，下游任务可能无法负担。同时，由于隐私问题，医学数据无法共享给第三方进行微调。为应对这些挑战，我们首次提出了一种针对提示驱动型医学图像分割的黑盒自适应技术——BAPS。BAPS包含两个组件：(i) 图像-提示解码器（IP解码器）模块，该模块根据给定图像和提示生成视觉提示；(ii) 零阶优化方法SPSA-GC，用于更新IP解码器而无需通过基础模型反向传播。因此，我们的方法无需了解基础模型的权重或梯度信息。我们在四种不同模态的数据上测试BAPS，结果表明该方法可将原始模型的性能提升约4%。