Textual scene graph parsing has become increasingly important in various vision-language applications, including image caption evaluation and image retrieval. However, existing scene graph parsers that convert image captions into scene graphs often suffer from two types of errors. First, the generated scene graphs fail to capture the true semantics of the captions or the corresponding images, resulting in a lack of faithfulness. Second, the generated scene graphs have high inconsistency, with the same semantics represented by different annotations. To address these challenges, we propose a novel dataset, which involves re-annotating the captions in Visual Genome (VG) using a new intermediate representation called FACTUAL-MR. FACTUAL-MR can be directly converted into faithful and consistent scene graph annotations. Our experimental results clearly demonstrate that the parser trained on our dataset outperforms existing approaches in terms of faithfulness and consistency. This improvement leads to a significant performance boost in both image caption evaluation and zero-shot image retrieval tasks. Furthermore, we introduce a novel metric for measuring scene graph similarity, which, when combined with the improved scene graph parser, achieves state-of-the-art (SOTA) results on multiple benchmark datasets for the aforementioned tasks. The code and dataset are available at https://github.com/zhuang-li/FACTUAL .

翻译：文本场景图解析在多种视觉-语言应用中日益重要，包括图像描述评估和图像检索。然而，现有将图像描述转换为场景图的解析器通常存在两类错误：首先，生成的场景图未能捕捉描述文本或对应图像的真实语义，导致缺乏忠实性；其次，生成的场景图存在高度不一致性，相同语义由不同标注表示。为解决这些问题，我们提出一种新数据集，利用名为FACTUAL-MR的新型中间表示对视觉基因组（VG）中的描述文本进行重新标注。FACTUAL-MR可直接转换为忠实且一致的场景图标注。实验结果表明，基于我们数据集训练的解析器在忠实性和一致性方面均优于现有方法。这一改进显著提升了图像描述评估和零样本图像检索任务的性能。此外，我们引入一种新的场景图相似度度量指标，该指标与改进的场景图解析器相结合，在多个上述任务的基准数据集上取得了最先进（SOTA）的结果。代码与数据集已发布于https://github.com/zhuang-li/FACTUAL。