Understanding the reasoning capabilities of Multimodal Large Language Models (MLLMs) is an important area of research. In this study, we introduce a dynamic benchmark, NPHardEval4V, aimed at addressing the existing gaps in evaluating the pure reasoning abilities of MLLMs. Our benchmark aims to provide a venue to disentangle the effect of various factors such as image recognition and instruction following, from the overall performance of the models, allowing us to focus solely on evaluating their reasoning abilities. It is built by converting textual description of questions from NPHardEval to image representations. Our findings reveal significant discrepancies in reasoning abilities across different models and highlight the relatively weak performance of MLLMs compared to LLMs in terms of reasoning. We also investigate the impact of different prompting styles, including visual, text, and combined visual and text prompts, on the reasoning abilities of MLLMs, demonstrating the different impacts of multimodal inputs in model performance. Unlike traditional benchmarks, which focus primarily on static evaluations, our benchmark will be updated monthly to prevent overfitting and ensure a more authentic and fine-grained evaluation of the models. We believe that this benchmark can aid in understanding and guide the further development of reasoning abilities in MLLMs. The benchmark dataset and code are available at https://github.com/lizhouf/NPHardEval4V

翻译：理解多模态大语言模型（MLLMs）的推理能力是一个重要的研究方向。本研究提出了一种动态基准——NPHardEval4V，旨在填补现有评估MLLMs纯推理能力的空白。该基准通过将NPHardEval中问题的文本描述转换为图像表示，构建了一个能够将图像识别、指令遵循等因素从模型整体性能中剥离的平台，从而专注于评估其推理能力。研究结果揭示了不同模型在推理能力上的显著差异，并指出MLLMs在推理方面相较于LLMs表现较弱。此外，我们还探究了不同提示风格（包括纯视觉提示、纯文本提示以及视觉与文本组合提示）对MLLMs推理能力的影响，验证了多模态输入对模型性能产生的差异作用。与传统静态评估基准不同，本基准将每月更新以防止过拟合，从而实现更真实、更细粒度的模型评估。我们相信，该基准有助于理解并指导MLLMs推理能力的进一步发展。基准数据集和代码已发布于 https://github.com/lizhouf/NPHardEval4V。