The success of large language models has inspired the computer vision community to explore image segmentation foundation model that is able to zero/few-shot generalize through prompt engineering. Segment-Anything(SAM), among others, is the state-of-the-art image segmentation foundation model demonstrating strong zero/few-shot generalization. Despite the success, recent studies reveal the weakness of SAM under strong distribution shift. In particular, SAM performs awkwardly on corrupted natural images, camouflaged images, medical images, etc. Motivated by the observations, we aim to develop a self-training based strategy to adapt SAM to target distribution. Given the unique challenges of large source dataset, high computation cost and incorrect pseudo label, we propose a weakly supervised self-training architecture with anchor regularization and low-rank finetuning to improve the robustness and computation efficiency of adaptation. We validate the effectiveness on 5 types of downstream segmentation tasks including natural clean/corrupted images, medical images, camouflaged images and robotic images. Our proposed method is task-agnostic in nature and outperforms pre-trained SAM and state-of-the-art domain adaptation methods on almost all downstream tasks with the same testing prompt inputs.

翻译：大语言模型取得的成功激励了计算机视觉领域探索能够通过提示工程实现零样本/少样本泛化的图像分割基础模型。Segment-Anything（SAM）作为其中最具代表性的模型，展现了强大的零样本/少样本泛化能力。尽管取得了成功，近期研究揭示了SAM在强分布偏移下的局限性，尤其是在受污染的自然图像、伪装图像、医学图像等场景中表现欠佳。基于这些观察，我们旨在开发一种基于自训练的策略，使SAM能够适应目标数据分布。针对源数据集庞大、计算成本高昂以及伪标签不准确等独特挑战，我们提出了一种弱监督自训练架构，结合锚点正则化和低秩微调技术，以提升自适应的鲁棒性和计算效率。我们在五种下游分割任务上验证了方法的有效性，包括自然干净/受污染图像、医学图像、伪装图像以及机器人视觉图像。所提方法本质上是任务无关的，在相同测试提示输入条件下，几乎在所有下游任务中均优于预训练SAM及当前最优的域自适应方法。