Vision-Language Pretraining (VLP) models have recently successfully facilitated many cross-modal downstream tasks. Most existing works evaluated their systems by comparing the fine-tuned downstream task performance. However, only average downstream task accuracy provides little information about the pros and cons of each VLP method, let alone provides insights on how the community can improve the systems in the future. Inspired by the CheckList for testing natural language processing, we exploit VL-CheckList, a novel framework to understand the capabilities of VLP models. The proposed method divides the image-texting ability of a VLP model into three categories: objects, attributes, and relations, and uses a novel taxonomy to further break down these three aspects. We conduct comprehensive studies to analyze seven recently popular VLP models via the proposed framework. Results confirm the effectiveness of the proposed method by revealing fine-grained differences among the compared models that were not visible from downstream task-only evaluation. Further results show promising research direction in building better VLP models. Our data and code are available at: https://github.com/om-ai-lab/VL-CheckList.

翻译：视觉语言预训练（VLP）模型近期在多项跨模态下游任务中取得了成功。现有研究大多通过比较微调后的下游任务性能来评估模型。然而，仅依赖下游任务的平均准确率难以揭示各VLP方法的优劣，更无法为社区未来改进系统提供指导。受自然语言处理中CheckList测试方法的启发，我们提出VL-CheckList这一新框架来理解VLP模型的能力。该方法将VLP模型的图像-文本理解能力划分为三类：目标、属性与关系，并通过创新的分类体系进一步分解这三个维度。我们利用该框架对七个近期流行的VLP模型开展全面分析。结果表明，所提方法能揭示仅通过下游任务评估无法观察到的模型间细粒度差异，从而验证了其有效性。进一步实验展示了构建更优VLP模型的有前景的研究方向。数据和代码已开源于：https://github.com/om-ai-lab/VL-CheckList。