Large vision-language models have recently achieved remarkable progress, exhibiting great perception and reasoning abilities concerning visual information. However, how to effectively evaluate these large vision-language models remains a major obstacle, hindering future model development. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but suffer from a lack of fine-grained ability assessment and non-robust evaluation metrics. Recent subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, but they are not scalable and display significant bias. In response to these challenges, we propose MMBench, a novel multi-modality benchmark. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of two elements. The first element is a meticulously curated dataset that surpasses existing similar benchmarks in terms of the number and variety of evaluation questions and abilities. The second element introduces a novel CircularEval strategy and incorporates the use of ChatGPT. This implementation is designed to convert free-form predictions into pre-defined choices, thereby facilitating a more robust evaluation of the model's predictions. MMBench is a systematically-designed objective benchmark for robustly evaluating the various abilities of vision-language models. We hope MMBench will assist the research community in better evaluating their models and encourage future advancements in this domain. Project page: https://opencompass.org.cn/mmbench.

翻译：大型视觉语言模型近期取得了显著进展，展现出对视觉信息的卓越感知与推理能力。然而，如何有效评估这些大型视觉语言模型仍是一大挑战，阻碍了模型的进一步发展。传统基准测试（如VQAv2或COCO Caption）虽可提供量化性能指标，但存在细粒度能力评估缺失与评估指标鲁棒性不足的问题。近期主观基准测试（如OwlEval）通过引入人工评估能力模型的综合能力，但其可扩展性有限且存在显著偏差。针对这些挑战，我们提出MMBench——一种新颖的多模态基准测试。MMBench系统性地构建了完整的评估流程，主要由两大元素组成：第一是精心策划的数据集，其在评估问题的数量、多样性及能力维度上均超越现有同类基准；第二是引入创新的CircularEval策略，并结合ChatGPT的使用，将自由形式的预测转换为预定义选项，从而实现对模型预测的更鲁棒评估。MMBench是一款系统设计的客观基准，用于稳健评估视觉语言模型的多项能力。我们期望MMBench能帮助研究社区更好地评估模型，并推动该领域的未来发展。项目页面：https://opencompass.org.cn/mmbench。