Foundation models for segmentation such as the Segment Anything Model (SAM) family exhibit strong zero-shot performance, but remain vulnerable in shifted or limited-knowledge domains. This work investigates whether uncertainty quantification can mitigate such challenges and enhance model generalisability in a domain-agnostic manner. To this end, we (1) curate UncertSAM, a benchmark comprising eight datasets designed to stress-test SAM under challenging segmentation conditions including shadows, transparency, and camouflage; (2) evaluate a suite of lightweight, post-hoc uncertainty estimation methods; and (3) assess a preliminary uncertainty-guided prediction refinement step. Among evaluated approaches, a last-layer Laplace approximation yields uncertainty estimates that correlate well with segmentation errors, indicating a meaningful signal. While refinement benefits are preliminary, our findings underscore the potential of incorporating uncertainty into segmentation models to support robust, domain-agnostic performance. Our benchmark and code are made publicly available.

翻译：以Segment Anything Model (SAM)系列为代表的分割基础模型展现出优异的零样本性能，但在分布偏移或知识受限的领域仍存在脆弱性。本研究探讨不确定性量化能否以领域无关的方式缓解此类挑战并提升模型泛化能力。为此，我们（1）构建了UncertSAM基准数据集，该数据集包含八个专门用于在阴影、透明及伪装等挑战性分割场景下压力测试SAM的数据集；（2）系统评估了一系列轻量级后验不确定性估计方法；（3）对初步设计的不确定性引导预测优化步骤进行了验证。在评估方法中，末层拉普拉斯近似法获得的不确定性估计与分割误差呈现良好相关性，表明其具备有效信号指示能力。尽管优化效果的提升尚属初步，但我们的研究结果凸显了将不确定性融入分割模型以支持鲁棒、领域无关性能的潜力。本研究的基准数据集与代码均已公开。