The Segment Anything Model (SAM) has gained significant attention for its impressive performance in image segmentation. However, it lacks proficiency in referring video object segmentation (RVOS) due to the need for precise user-interactive prompts and limited understanding of different modalities, such as language and vision. This paper presents the RefSAM model, which for the first time explores the potential of SAM for RVOS by incorporating multi-view information from diverse modalities and successive frames at different timestamps. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-Modal MLP that projects the text embedding of the referring expression into sparse and dense embeddings, serving as user-interactive prompts. Subsequently, a parameter-efficient tuning strategy is employed to effectively align and fuse the language and vision features. Through comprehensive ablation studies, we demonstrate the practical and effective design choices of our strategy. Extensive experiments conducted on Ref-Youtu-VOS and Ref-DAVIS17 datasets validate the superiority and effectiveness of our RefSAM model over existing methods. The code and models will be made publicly at \href{https://github.com/LancasterLi/RefSAM}{github.com/LancasterLi/RefSAM}.

翻译：分割万物模型（Segment Anything Model, SAM）因其在图像分割中的卓越性能而备受关注。然而，由于需要精确的用户交互式提示且对语言与视觉等多模态信息理解有限，SAM在指代视频目标分割（Referring Video Object Segmentation, RVOS）任务中表现不足。本文提出RefSAM模型，首次探索SAM在RVOS中的潜力，通过融合来自不同模态的多视角信息以及不同时间戳的连续帧。该方法通过采用轻量级跨模态多层感知机（Cross-Modal MLP）将指代表达的文本嵌入投射为稀疏与稠密嵌入，作为用户交互式提示，从而适配原始SAM模型以增强跨模态学习。随后，采用参数高效微调策略有效对齐并融合语言与视觉特征。通过全面消融研究，我们验证了策略中实用且有效的设计选择。在Ref-Youtu-VOS和Ref-DAVIS17数据集上的大量实验表明，我们的RefSAM模型相比现有方法具有优越性与有效性。代码与模型将开源至\href{https://github.com/LancasterLi/RefSAM}{github.com/LancasterLi/RefSAM}。