We examine the recent Segment Anything Model (SAM) on medical images, and report both quantitative and qualitative zero-shot segmentation results on nine medical image segmentation benchmarks, covering various imaging modalities, such as optical coherence tomography (OCT), magnetic resonance imaging (MRI), and computed tomography (CT), as well as different applications including dermatology, ophthalmology, and radiology. Our experiments reveal that while SAM demonstrates stunning segmentation performance on images from the general domain, for those out-of-distribution images, e.g., medical images, its zero-shot segmentation performance is still limited. Furthermore, SAM demonstrated varying zero-shot segmentation performance across different unseen medical domains. For example, it had a 0.8704 mean Dice score on segmenting under-bruch's membrane layer of retinal OCT, whereas the segmentation accuracy drops to 0.0688 when segmenting retinal pigment epithelium. For certain structured targets, e.g., blood vessels, the zero-shot segmentation of SAM completely failed, whereas a simple fine-tuning of it with small amount of data could lead to remarkable improvements of the segmentation quality. Our study indicates the versatility of generalist vision foundation models on solving specific tasks in medical imaging, and their great potential to achieve desired performance through fine-turning and eventually tackle the challenges of accessing large diverse medical datasets and the complexity of medical domains.

翻译：我们研究了最新的Segment Anything模型（SAM）在医学图像上的表现，报告了在九个医学图像分割基准上的定性与定量零样本分割结果，涵盖多种成像模态，如光学相干断层扫描（OCT）、磁共振成像（MRI）和计算机断层扫描（CT），以及包括皮肤科、眼科和放射科在内的不同应用领域。实验表明，尽管SAM在通用领域图像上展现出惊人的分割性能，但对于分布外图像（如医学图像），其零样本分割能力仍较为有限。此外，SAM在不同未见医学领域中的零样本分割表现存在差异。例如，在视网膜OCT图像的Bruch膜下层分割中，其平均Dice系数达到0.8704，而在视网膜色素上皮层分割中准确率骤降至0.0688。对于某些结构化目标（如血管），SAM的零样本分割完全失效，而使用少量数据对其进行简单微调即可显著提升分割质量。本研究揭示了通用视觉基础模型在解决医学成像特定任务中的多功能性，以及通过微调实现预期性能的巨大潜力，最终有助于应对获取大规模多样化医学数据集的挑战以及医学领域的复杂性。