Recent advancements in Multi-modal Large Language Models (MLLMs) have significantly improved their performance in tasks combining vision and language. However, challenges persist in detailed multi-modal understanding, comprehension of complex tasks, and reasoning over multi-modal information. This paper introduces MMCTAgent, a novel multi-modal critical thinking agent framework designed to address the inherent limitations of current MLLMs in complex visual reasoning tasks. Inspired by human cognitive processes and critical thinking, MMCTAgent iteratively analyzes multi-modal information, decomposes queries, plans strategies, and dynamically evolves its reasoning. Additionally, MMCTAgent incorporates critical thinking elements such as verification of final answers and self-reflection through a novel approach that defines a vision-based critic and identifies task-specific evaluation criteria, thereby enhancing its decision-making abilities. Through rigorous evaluations across various image and video understanding benchmarks, we demonstrate that MMCTAgent (with and without the critic) outperforms both foundational MLLMs and other tool-augmented pipelines.

翻译：近年来，多模态大语言模型在结合视觉与语言的任务中表现显著提升。然而，其在细致的多模态理解、复杂任务解析以及多模态信息推理方面仍面临挑战。本文提出MMCTAgent，一种新颖的多模态批判性思维智能体框架，旨在解决当前MLLMs在复杂视觉推理任务中的固有局限。受人类认知过程与批判性思维启发，MMCTAgent迭代分析多模态信息、分解查询、规划策略并动态演进其推理过程。此外，MMCTAgent通过一种创新方法整合了批判性思维要素，包括最终答案的验证与自我反思，该方法定义了基于视觉的评判器并识别任务特定的评估标准，从而增强了其决策能力。通过在多种图像与视频理解基准上的严格评估，我们证明MMCTAgent（无论是否包含评判器）均优于基础MLLMs及其他工具增强流程。