In this study, we aim to develop a model that comprehends a natural language instruction (e.g., "Go to the living room and get the nearest pillow to the radio art on the wall") and generates a segmentation mask for the target everyday object. The task is challenging because it requires (1) the understanding of the referring expressions for multiple objects in the instruction, (2) the prediction of the target phrase of the sentence among the multiple phrases, and (3) the generation of pixel-wise segmentation masks rather than bounding boxes. Studies have been conducted on languagebased segmentation methods; however, they sometimes mask irrelevant regions for complex sentences. In this paper, we propose the Multimodal Diffusion Segmentation Model (MDSM), which generates a mask in the first stage and refines it in the second stage. We introduce a crossmodal parallel feature extraction mechanism and extend diffusion probabilistic models to handle crossmodal features. To validate our model, we built a new dataset based on the well-known Matterport3D and REVERIE datasets. This dataset consists of instructions with complex referring expressions accompanied by real indoor environmental images that feature various target objects, in addition to pixel-wise segmentation masks. The performance of MDSM surpassed that of the baseline method by a large margin of +10.13 mean IoU.

翻译：本研究旨在开发一种模型，该模型能够理解自然语言指令（例如：“去客厅，取距离墙壁收音机艺术品最近的枕头”），并为日常目标物体生成分割掩码。该任务具有挑战性，原因在于需要：（1）理解指令中涉及多个物体的指代表述；（2）从多个短语中预测句子的目标短语；（3）生成像素级分割掩码，而非边界框。当前已有基于语言的分割方法研究，但这些方法在处理复杂句子时，有时会掩码不相关区域。本文提出多模态扩散分割模型（MDSM），该模型在第一阶段生成掩码，并在第二阶段对其进行优化。我们引入了跨模态并行特征提取机制，并扩展了扩散概率模型以处理跨模态特征。为验证模型有效性，我们基于著名的Matterport3D和REVERIE数据集构建了新数据集。该数据集包含带有复杂指代表述的指令、对应真实室内环境图像（涵盖多种目标物体），以及像素级分割掩码。MDSM的性能远超基线方法，平均交并比（mIoU）提升了+10.13个百分点。