Referring Image Segmentation (RIS) is a challenging task that requires an algorithm to segment objects referred by free-form language expressions. Despite significant progress in recent years, most state-of-the-art (SOTA) methods still suffer from considerable language-image modality gap at the pixel and word level. These methods generally 1) rely on sentence-level language features for language-image alignment and 2) lack explicit training supervision for fine-grained visual grounding. Consequently, they exhibit weak object-level correspondence between visual and language features. Without well-grounded features, prior methods struggle to understand complex expressions that require strong reasoning over relationships among multiple objects, especially when dealing with rarely used or ambiguous clauses. To tackle this challenge, we introduce a novel Mask Grounding auxiliary task that significantly improves visual grounding within language features, by explicitly teaching the model to learn fine-grained correspondence between masked textual tokens and their matching visual objects. Mask Grounding can be directly used on prior RIS methods and consistently bring improvements. Furthermore, to holistically address the modality gap, we also design a cross-modal alignment loss and an accompanying alignment module. These additions work synergistically with Mask Grounding. With all these techniques, our comprehensive approach culminates in MagNet Mask-grounded Network), an architecture that significantly outperforms prior arts on three key benchmarks (RefCOCO, RefCOCO+ and G-Ref), demonstrating our method's effectiveness in addressing current limitations of RIS algorithms. Our code and pre-trained weights will be released.

翻译：指代图像分割（RIS）是一项具有挑战性的任务，要求算法能够分割由自由形式语言表达所指代的目标。尽管近年来取得了显著进展，但大多数最先进（SOTA）方法在像素和词汇层面仍存在较大的语言-图像模态差异。这些方法通常：1）依赖句子级语言特征进行语言-图像对齐，2）缺乏针对细粒度视觉定位的显式训练监督。因此，它们在视觉特征与语言特征之间的目标级对应关系上表现薄弱。若缺乏良好对齐的特征，现有方法难以理解需要对多个物体间关系进行强推理的复杂表达，尤其是在处理罕见或歧义性从句时。为解决这一挑战，我们提出了一种新颖的掩码引导辅助任务，通过显式教导模型学习掩码文本标记与其对应视觉目标之间的细粒度对应关系，显著提升语言特征内的视觉定位能力。掩码引导可直接应用于现有RIS方法并持续带来性能提升。此外，为全面解决模态差异问题，我们还设计了跨模态对齐损失函数及配套的对齐模块。这些增强与掩码引导协同作用。基于上述技术，我们提出完整方案MagNet（掩码引导网络），该架构在三个关键基准（RefCOCO、RefCOCO+和G-Ref）上显著优于先前方法，充分证明了我们在解决当前RIS算法局限性方面的有效性。我们的代码与预训练权重将公开发布。