The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It designed a novel promotable segmentation task, ensuring zero-shot image segmentation using the pre-trained model via two main modes including automatic everything and manual prompt. SAM has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging due to the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. Meanwhile, zero-shot and efficient MIS can well reduce the annotation time and boost the development of medical image analysis. Hence, SAM seems to be a potential tool and its performance on large medical datasets should be further validated. We collected and sorted 52 open-source datasets, and built a large medical segmentation dataset with 16 modalities, 68 objects, and 553K slices. We conducted a comprehensive analysis of different SAM testing strategies on the so-called COSMOS 553K dataset. Extensive experiments validate that SAM performs better with manual hints like points and boxes for object perception in medical images, leading to better performance in prompt mode compared to everything mode. Additionally, SAM shows remarkable performance in some specific objects and modalities, but is imperfect or even totally fails in other situations. Finally, we analyze the influence of different factors (e.g., the Fourier-based boundary complexity and size of the segmented objects) on SAM's segmentation performance. Extensive experiments validate that SAM's zero-shot segmentation capability is not sufficient to ensure its direct application to the MIS.

翻译：通用图像分割模型（SAM）是首个用于通用图像分割的基础模型。它设计了一种新颖的可提示分割任务，通过自动万物分割和人工提示两种模式，利用预训练模型实现零样本图像分割。SAM在各类自然图像分割任务中取得了令人瞩目的成果。然而，医学图像分割因模态复杂、解剖结构精细、物体边界不确定且复杂、以及物体尺度范围广泛而更具挑战性。同时，零样本高效医学图像分割可大幅减少标注时间并推动医学图像分析发展。因此，SAM作为潜在工具，其在大规模医学数据集上的表现有待进一步验证。我们收集整理了52个开源数据集，构建了包含16种模态、68个目标、55.3万张切片的大规模医学分割数据集（COSMOS 553K）。基于该数据集，我们系统分析了SAM在不同测试策略下的表现。大量实验表明，SAM在医学图像中通过点、框等人工提示进行目标感知时表现更优，提示模式的效果优于万物分割模式。此外，SAM在特定目标与模态中展现出显著性能，但在其他情况下存在性能不足甚至完全失效的问题。最后，我们分析了傅里叶边界复杂度、分割目标尺寸等不同因素对SAM分割性能的影响。综合实验证实，SAM的零样本分割能力尚不足以直接应用于医学图像分割。