We propose a margin-based loss for tuning joint vision-language models so that their gradient-based explanations are consistent with region-level annotations provided by humans for relatively smaller grounding datasets. We refer to this objective as Attention Mask Consistency (AMC) and demonstrate that it produces superior visual grounding results than previous methods that rely on using vision-language models to score the outputs of object detectors. Particularly, a model trained with AMC on top of standard vision-language modeling objectives obtains a state-of-the-art accuracy of 86.49% in the Flickr30k visual grounding benchmark, an absolute improvement of 5.38% when compared to the best previous model trained under the same level of supervision. Our approach also performs exceedingly well on established benchmarks for referring expression comprehension where it obtains 80.34% accuracy in the easy test of RefCOCO+, and 64.55% in the difficult split. AMC is effective, easy to implement, and is general as it can be adopted by any vision-language model, and can use any type of region annotations.

翻译：我们提出一种基于边界的损失函数，用于微调联合视觉-语言模型，使其基于梯度的解释与人类在相对较小的定位数据集上提供的区域级标注保持一致。我们将这一目标称为注意力掩码一致性（AMC），并证明其相较于依赖视觉-语言模型对目标检测器输出进行评分的先前方法，能产生更优的视觉定位结果。具体而言，在标准视觉-语言建模目标基础上使用AMC训练的模型，在Flickr30k视觉定位基准测试中达到86.49%的先进准确率，相较于在相同监督水平下训练的最佳先前模型，绝对提升达5.38%。我们的方法在指代表达式理解的既定基准测试中同样表现卓越：在RefCOCO+简单测试集上获得80.34%的准确率，在困难子集上获得64.55%。AMC高效、易于实现且具有通用性——可被任何视觉-语言模型采用，并能处理任意类型的区域标注。