Referring image segmentation (RIS) is a fundamental vision-language task that intends to segment a desired object from an image based on a given natural language expression. Due to the essentially distinct data properties between image and text, most of existing methods either introduce complex designs towards fine-grained vision-language alignment or lack required dense alignment, resulting in scalability issues or mis-segmentation problems such as over- or under-segmentation. To achieve effective and efficient fine-grained feature alignment in the RIS task, we explore the potential of masked multimodal modeling coupled with self-distillation and propose a novel cross-modality masked self-distillation framework named CM-MaskSD, in which our method inherits the transferred knowledge of image-text semantic alignment from CLIP model to realize fine-grained patch-word feature alignment for better segmentation accuracy. Moreover, our CM-MaskSD framework can considerably boost model performance in a nearly parameter-free manner, since it shares weights between the main segmentation branch and the introduced masked self-distillation branches, and solely introduces negligible parameters for coordinating the multimodal features. Comprehensive experiments on three benchmark datasets (i.e. RefCOCO, RefCOCO+, G-Ref) for the RIS task convincingly demonstrate the superiority of our proposed framework over previous state-of-the-art methods.

翻译：指代图像分割（RIS）是一项基础的视觉-语言任务，旨在根据给定的自然语言表达从图像中分割出目标物体。由于图像与文本本质上存在截然不同的数据特性，现有方法或需引入复杂设计以实现细粒度的视觉-语言对齐，或缺乏所需的密集对齐，导致可扩展性问题及误分割（如过度分割或欠分割）。为实现RIS任务中高效且有效的细粒度特征对齐，本文探索了掩码多模态建模与自蒸馏结合的潜力，提出了一种新颖的跨模态掩码自蒸馏框架——CM-MaskSD。该方法继承自CLIP模型的图像-文本语义对齐迁移知识，实现了细粒度的补丁-词语特征对齐，从而提升分割精度。此外，CM-MaskSD框架能以近乎无参数的方式显著提升模型性能：主分割分支与引入的掩码自蒸馏分支共享权重，仅引入可忽略的参数用于协调多模态特征。在三个RIS任务基准数据集（RefCOCO、RefCOCO+、G-Ref）上的全面实验令人信服地证明了所提框架相较于先前最先进方法的优越性。