Widely adopted medical image segmentation methods, although efficient, are primarily deterministic and remain poorly amenable to natural language prompts. Thus, they lack the capability to estimate multiple proposals, human interaction, and cross-modality adaptation. Recently, text-to-image diffusion models have shown potential to bridge the gap. However, training them from scratch requires a large dataset-a limitation for medical image segmentation. Furthermore, they are often limited to binary segmentation and cannot be conditioned on a natural language prompt. To this end, we propose a novel framework called ProGiDiff that leverages existing image generation models for medical image segmentation purposes. Specifically, we propose a ControlNet-style conditioning mechanism with a custom encoder, suitable for image conditioning, to steer a pre-trained diffusion model to output segmentation masks. It naturally extends to a multi-class setting simply by prompting the target organ. Our experiment on organ segmentation from CT images demonstrates strong performance compared to previous methods and could greatly benefit from an expert-in-the-loop setting to leverage multiple proposals. Importantly, we demonstrate that the learned conditioning mechanism can be easily transferred through low-rank, few-shot adaptation to segment MR images.

翻译：广泛采用的医学图像分割方法虽然高效，但主要是确定性的，且难以适应自然语言提示。因此，它们缺乏估计多个候选方案、进行人机交互以及实现跨模态适应的能力。最近，文本到图像扩散模型显示出弥合这一差距的潜力。然而，从头开始训练这些模型需要大规模数据集——这在医学图像分割领域是一个限制。此外，它们通常仅限于二元分割，并且无法以自然语言提示为条件。为此，我们提出了一种名为ProGiDiff的新型框架，该框架利用现有的图像生成模型实现医学图像分割。具体而言，我们提出了一种采用定制编码器的ControlNet风格条件机制，该机制适用于图像条件控制，能够引导预训练的扩散模型输出分割掩码。通过简单地提示目标器官，该方法可自然地扩展到多类别设置。我们在CT图像器官分割上的实验表明，与先前方法相比，该方法表现出强大的性能，并且能够极大地受益于专家在环设置以利用多个候选方案。重要的是，我们证明了所学习的条件机制可以通过低秩、少样本适应轻松迁移，以分割MR图像。