The robustness of multimodal deep learning models to realistic changes in the input text is critical for their applicability to important tasks such as text-to-image retrieval and cross-modal entailment. To measure robustness, several existing approaches edit the text data, but do so without leveraging the cross-modal information present in multimodal data. Information from the visual modality, such as color, size, and shape, provide additional attributes that users can include in their inputs. Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., "girl on a chair" to "little girl on a wooden chair"). Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic. We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and cross-modal entailment to perform poorly, resulting in relative drops of 15% in MRR and 20% in $F_1$ score, respectively. Crowd-sourced annotations demonstrate that cross-modal insertions lead to higher quality augmentations for multimodal data than augmentations using text-only data, and are equivalent in quality to original examples. We release the code to encourage robustness evaluations of deep vision-and-language models: https://github.com/claws-lab/multimodal-robustness-xmai.

翻译：多模态深度学习模型对输入文本实际变化的鲁棒性，对于其在文本到图像检索和跨模态蕴含等重要任务中的适用性至关重要。为衡量鲁棒性，现有的一些方法对文本数据进行编辑，但并未利用多模态数据中存在的跨模态信息。视觉模态中的信息（如颜色、大小和形状）提供了用户可纳入其输入的额外属性。因此，我们提出跨模态属性插入作为一种针对视觉-语言数据的现实扰动策略，该方法将图像中物体的视觉属性插入到对应的文本中（例如，将“女孩坐在椅子上”变为“小女孩坐在木椅上”）。我们提出的跨模态属性插入方法具有模块化、可控且任务无关的特点。研究发现，使用跨模态插入增强输入文本会导致文本到图像检索和跨模态蕴含的最先进方法性能显著下降，在MRR和$F_1$分数上分别相对下降15%和20%。众包标注结果表明，与仅使用文本数据的增强相比，跨模态插入能为多模态数据生成更高质量的增强，且质量与原始示例相当。我们公开发布代码以鼓励对深度视觉-语言模型的鲁棒性评估：https://github.com/claws-lab/multimodal-robustness-xmai。