While large multi-modal models (LMMs) have exhibited impressive capabilities across diverse tasks, their effectiveness in handling complex tasks has been limited by the prevailing single-step reasoning paradigm. To this end, this paper proposes VoCoT, a multi-step Visually grounded object-centric Chain-of-Thought reasoning framework tailored for inference with LMMs. VoCoT is characterized by two key features: (1) object-centric reasoning paths that revolve around cross-modal shared object-level information, and (2) visually grounded representation of object concepts in a multi-modal interleaved and aligned manner, which effectively bridges the modality gap within LMMs during long-term generation. Additionally, we construct an instruction dataset to facilitate LMMs in adapting to reasoning with VoCoT. By introducing VoCoT into the prevalent open-source LMM architecture, we introduce VolCano. With only 7B parameters and limited input resolution, VolCano demonstrates excellent performance across various scenarios, surpassing SOTA models, including GPT-4V, in tasks requiring complex reasoning. Our code, data and model will be available at https://github.com/RupertLuo/VoCoT.

翻译：尽管大型多模态模型（LMMs）已在多种任务中展现出令人印象深刻的能力，但其处理复杂任务的有效性仍受限于当前主流的单步推理范式。为此，本文提出VoCoT，一种专为LMM推理设计的多步视觉基础对象中心思维链推理框架。VoCoT具有两个关键特征：（1）围绕跨模态共享的对象级信息构建的对象中心推理路径；（2）以多模态交错对齐的方式对对象概念进行视觉基础表示，从而在长序列生成过程中有效弥合LMM内部的模态鸿沟。此外，我们构建了一个指令数据集，以促进LMM适应VoCoT推理方式。通过将VoCoT引入当前主流的开源LMM架构，我们提出了VolCano模型。仅凭70亿参数和有限的输入分辨率，VolCano在多种场景中均表现出卓越性能，在需要复杂推理的任务中超越了包括GPT-4V在内的最先进模型。我们的代码、数据及模型将在https://github.com/RupertLuo/VoCoT 公开。