Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at https://core-mm.github.io/

翻译：多模态大语言模型（MLLMs）在人工智能领域日益凸显。这类模型不仅在传统视觉-语言任务中表现出色，还在当代多模态基准测试中展现出令人瞩目的性能。尽管许多基准测试试图全面评估MLLMs，但它们通常聚焦于基础推理任务，往往仅产生简单的“是/否”或多选响应。这种方法自然会导致混淆，难以明确判定MLLMs的推理能力。为缓解此问题，我们手动构建了一个专为MLLMs设计的基准数据集，侧重于复杂推理任务。该基准涵盖三大关键推理类别：演绎推理、溯因推理和类比推理。我们特意设计数据集中的查询，以在生成答案的过程中激发MLLMs的推理能力。为了在多种MLLMs之间进行公平比较，我们将中间推理步骤纳入评估标准。当MLLM无法给出确定性答案时，我们通过要求其提供中间推理步骤来评估其推理能力。若这些步骤与人工标注一致，则赋予相应分数。此评估方案类似于人类评估中常用的方法（如考试或作业），我们认为它相较于现有基准是一种更有效的评估技术。我们使用这一经严格开发的开放式多步精细推理基准，评估了若干代表性MLLMs，旨在挑战并准确测量它们的推理能力。代码与数据将发布于 https://core-mm.github.io/