With the success of Large Language Models (LLMs), a surge of Generative Vision-Language Models (GVLMs) have been constructed via multimodal instruction tuning. The tuning recipe substantially deviates from the common contrastive vision-language learning. However, the performance of GVLMs in multimodal compositional reasoning remains largely unexplored, as existing evaluation metrics and benchmarks focus predominantly on assessing contrastive models like CLIP. In this paper, we examine the potential evaluation metrics to assess the GVLMs and hypothesize generative score methods are suitable for evaluating compositionality. In addition, current benchmarks tend to prioritize syntactic correctness over semantics. The presence of morphological bias in these benchmarks can be exploited by GVLMs, leading to ineffective evaluations. To combat this, we define a MorphoBias Score to quantify the morphological bias and propose a novel LLM-based strategy to calibrate the bias. Moreover, a challenging task is added to evaluate the robustness of GVLMs against inherent inclination toward syntactic correctness. We include the calibrated dataset and the task into a new benchmark, namely MOrphologicall De-biased Benchmark (MODE). Our study provides the first unbiased benchmark for the compositionality of GVLMs, facilitating future research in this direction. We will release our code and datasets.

翻译：随着大语言模型（LLM）的成功，通过多模态指令微调涌现出大量生成式视觉语言模型（GVLM）。该微调方法显著不同于常见的对比式视觉语言学习。然而，GVLM在多模态组合推理中的性能仍缺乏系统研究，现有评估指标和基准主要聚焦于评估CLIP等对比式模型。本文系统考察了评估GVLM的潜在指标，提出生成式评分方法适用于衡量组合性。此外，现有基准倾向于优先考虑句法正确性而非语义正确性，GVLM可利用此类基准中的形态偏误导致评估失效。为解决该问题，我们定义了量化形态偏误的MorphoBias Score，并提出基于LLM的新型校准策略。进一步，我们构建了挑战性任务以评估GVLM对句法正确性固有倾向的鲁棒性。我们将校准数据集与任务整合为新型基准——形态学去偏基准（MODE）。本研究首次为GVLM的组合性提供了无偏基准，将推动该方向的研究进展。我们将在后续开源代码与数据集。