In this paper, we address the challenge of image resolution variation for the Segment Anything Model (SAM). SAM, known for its zero-shot generalizability, exhibits a performance degradation when faced with datasets with varying image sizes. Previous approaches tend to resize the image to a fixed size or adopt structure modifications, hindering the preservation of SAM's rich prior knowledge. Besides, such task-specific tuning necessitates a complete retraining of the model, which is cost-expensive and unacceptable for deployment in the downstream tasks. In this paper, we reformulate this issue as a length extrapolation problem, where token sequence length varies while maintaining a consistent patch size for images of different sizes. To this end, we propose Scalable Bias-Mode Attention Mask (BA-SAM) to enhance SAM's adaptability to varying image resolutions while eliminating the need for structure modifications. Firstly, we introduce a new scaling factor to ensure consistent magnitude in the attention layer's dot product values when the token sequence length changes. Secondly, we present a bias-mode attention mask that allows each token to prioritize neighboring information, mitigating the impact of untrained distant information. Our BA-SAM demonstrates efficacy in two scenarios: zero-shot and fine-tuning. Extensive evaluation on diverse datasets, including DIS5K, DUTS, ISIC, COD10K, and COCO, reveals its ability to significantly mitigate performance degradation in the zero-shot setting and achieve state-of-the-art performance with minimal fine-tuning. Furthermore, we propose a generalized model and benchmark, showcasing BA-SAM's generalizability across all four datasets simultaneously.

翻译：本文针对Segment Anything Model（SAM）在图像分辨率变化场景下的挑战展开研究。以零样本泛化能力著称的SAM，在处理不同尺寸图像数据集时会出现性能退化。现有方法通常将图像调整为固定尺寸或采用结构修改策略，这阻碍了SAM丰富先验知识的保留。此外，此类任务特定微调需要对模型进行完全重新训练，在下游任务部署中成本高昂且难以接受。本文将该问题重构为长度外推问题——在保持图像不同尺寸下补丁尺寸一致的同时，令令牌序列长度动态变化。为此，我们提出可扩展偏置模式注意力掩码（BA-SAM），在不修改模型结构的前提下增强SAM对可变图像分辨率的适应性。首先引入新型缩放因子，确保令牌序列长度变化时注意力层点积值的量级保持一致；随后提出偏置模式注意力掩码，使每个令牌优先关注邻近信息，缓解未经训练的远距离信息干扰。BA-SAM在零样本和微调两个场景中均展现出有效性。在DIS5K、DUTS、ISIC、COD10K和COCO等多个数据集上的广泛评估表明，该机制能在零样本设置下显著缓解性能退化，并通过最小化微调实现最先进性能。进一步地，我们提出通用模型与基准测试，证明BA-SAM在全部四个数据集上同时具备跨数据集泛化能力。