Large vision-language models have recently achieved remarkable progress, exhibiting great perception and reasoning abilities concerning visual information. However, how to effectively evaluate these large vision-language models remains a major obstacle, hindering future model development. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but suffer from a lack of fine-grained ability assessment and non-robust evaluation metrics. Recent subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, but they are not scalable and display significant bias. In response to these challenges, we propose MMBench, a novel multi-modality benchmark. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of two elements. The first element is a meticulously curated dataset that surpasses existing similar benchmarks in terms of the number and variety of evaluation questions and abilities. The second element introduces a novel CircularEval strategy and incorporates the use of ChatGPT. This implementation is designed to convert free-form predictions into pre-defined choices, thereby facilitating a more robust evaluation of the model's predictions. MMBench is a systematically-designed objective benchmark for robustly evaluating the various abilities of vision-language models. We hope MMBench will assist the research community in better evaluating their models and encourage future advancements in this domain. Project page: https://opencompass.org.cn/mmbench.

翻译：大型视觉语言模型近期取得了显著进展，展现出对视觉信息的出色感知与推理能力。然而，如何有效评估这些大型视觉语言模型仍是主要障碍，阻碍了未来模型的发展。传统基准如VQAv2或COCO Caption虽能提供定量性能指标，但存在缺乏细粒度能力评估和评估指标鲁棒性不足的问题。近期主观基准（如OwlEval）通过引入人工评估实现了对模型能力的全面评价，但存在可扩展性差且偏差显著的问题。针对这些挑战，我们提出新型多模态基准MMBench。MMBench系统化地构建了全面评估流程，主要包含两个要素：其一是精心策划的数据集，在评估问题的数量、多样性和能力覆盖上超越现有同类基准；其二是引入创新的CircularEval策略，并整合ChatGPT应用，旨在将自由形式预测转化为预定义选项，从而提升模型预测评估的鲁棒性。MMBench是为稳健评估视觉语言模型多种能力而系统设计的客观基准。我们期待MMBench能帮助研究社区更有效地评估其模型，并推动该领域的未来发展。项目页面：https://opencompass.org.cn/mmbench。