Accurate segmentation of polyps and skin lesions is essential for diagnosing colorectal and skin cancers. While various segmentation methods for polyps and skin lesions using fully supervised deep learning techniques have been developed, the pixel-level annotation of medical images by doctors is both time-consuming and costly. Foundational vision models like the Segment Anything Model (SAM) have demonstrated superior performance; however, directly applying SAM to medical segmentation may not yield satisfactory results due to the lack of domain-specific medical knowledge. In this paper, we propose BiSeg-SAM, a SAM-guided weakly supervised prompting and boundary refinement network for the segmentation of polyps and skin lesions. Specifically, we fine-tune SAM combined with a CNN module to learn local features. We introduce a WeakBox with two functions: automatically generating box prompts for the SAM model and using our proposed Multi-choice Mask-to-Box (MM2B) transformation for rough mask-to-box conversion, addressing the mismatch between coarse labels and precise predictions. Additionally, we apply scale consistency (SC) loss for prediction scale alignment. Our DetailRefine module enhances boundary precision and segmentation accuracy by refining coarse predictions using a limited amount of ground truth labels. This comprehensive approach enables BiSeg-SAM to achieve excellent multi-task segmentation performance. Our method demonstrates significant superiority over state-of-the-art (SOTA) methods when tested on five polyp datasets and one skin cancer dataset.

翻译：准确分割息肉和皮肤病变对于诊断结直肠癌和皮肤癌至关重要。尽管已开发出多种利用全监督深度学习技术进行息肉和皮肤病变分割的方法，但由医生对医学图像进行像素级标注既耗时又昂贵。基础视觉模型如Segment Anything Model (SAM)已展现出卓越性能；然而，由于缺乏领域特定的医学知识，直接将SAM应用于医学分割可能无法获得理想结果。本文提出BiSeg-SAM，一种用于息肉和皮肤病变分割的SAM引导弱监督提示与边界细化网络。具体而言，我们微调SAM并结合CNN模块以学习局部特征。我们引入具备双重功能的WeakBox：为SAM模型自动生成框提示，并利用我们提出的多选掩码到框（MM2B）转换进行粗略的掩码到框转换，以解决粗糙标签与精确预测之间的不匹配问题。此外，我们应用尺度一致性（SC）损失以实现预测尺度对齐。我们的DetailRefine模块通过使用有限量的真实标签细化粗糙预测，从而提升边界精度和分割准确性。这种综合方法使BiSeg-SAM能够实现出色的多任务分割性能。在五个息肉数据集和一个皮肤癌数据集上的测试表明，我们的方法相较于最先进（SOTA）方法具有显著优越性。