Referring image segmentation (RIS) is a fundamental vision-language task that intends to segment a desired object from an image based on a given natural language expression. Due to the essentially distinct data properties between image and text, most of existing methods either introduce complex designs towards fine-grained vision-language alignment or lack required dense alignment, resulting in scalability issues or mis-segmentation problems such as over- or under-segmentation. To achieve effective and efficient fine-grained feature alignment in the RIS task, we explore the potential of masked multimodal modeling coupled with self-distillation and propose a novel cross-modality masked self-distillation framework named CM-MaskSD, in which our method inherits the transferred knowledge of image-text semantic alignment from CLIP model to realize fine-grained patch-word feature alignment for better segmentation accuracy. Moreover, our CM-MaskSD framework can considerably boost model performance in a nearly parameter-free manner, since it shares weights between the main segmentation branch and the introduced masked self-distillation branches, and solely introduces negligible parameters for coordinating the multimodal features. Comprehensive experiments on three benchmark datasets (i.e. RefCOCO, RefCOCO+, G-Ref) for the RIS task convincingly demonstrate the superiority of our proposed framework over previous state-of-the-art methods.

翻译：摘要：指代图像分割（RIS）是一项基础的视觉-语言任务，旨在根据给定的自然语言表达从图像中分割出目标物体。由于图像与文本之间存在本质上的数据特性差异，现有方法要么引入复杂设计以实现细粒度的视觉-语言对齐，要么缺乏所需的密集对齐，从而引发可扩展性问题或误分割现象（如过度分割或欠分割）。为在RIS任务中实现高效且有效的细粒度特征对齐，我们探索了掩蔽多模态建模与自蒸馏相结合的潜力，并提出了一种新颖的跨模态掩蔽自蒸馏框架——CM-MaskSD。该方法继承了CLIP模型中图像-文本语义对齐的迁移知识，实现了细粒度的图像块-词语特征对齐，从而提升了分割精度。此外，CM-MaskSD框架几乎以无参方式显著提升模型性能——其主分割分支与引入的掩蔽自蒸馏分支共享权重，仅引入可忽略的参数用于协调多模态特征。在三个基准数据集（即RefCOCO、RefCOCO+、G-Ref）上的全面实验令人信服地证明了所提框架相对于先前最先进方法的优越性。