Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies are mostly isolated in the language modality with LLMs, where LLMs are hard to deploy. To elicit CoT reasoning in multimodality, a possible solution is to fine-tune small language models by fusing the vision and language features to perform CoT reasoning. The key challenge is that those language models tend to generate hallucinated reasoning chains that mislead the answer inference. To mitigate the effect of such mistakes, we propose Multimodal-CoT that incorporates vision features in a decoupled training framework. The framework separates the rationale generation and answer inference into two stages. By incorporating the vision features in both stages, the model is able to generate effective rationales that contribute to answer inference. With Multimodal-CoT, our model under 1 billion parameters outperforms the previous state-of-the-art LLM (GPT-3.5) by 16% (75.17%->91.68%) on the ScienceQA benchmark and even surpasses human performance. Code is publicly available at https://github.com/amazon-science/mm-cot.

翻译：大型语言模型（LLMs）通过利用思维链（CoT）提示生成中间推理链作为推断答案的依据，在复杂推理任务上展现出显著性能。然而，现有CoT研究大多局限于LLMs的语言模态，且LLMs难以部署。为激发多模态场景下的CoT推理，一种可行方案是通过融合视觉与语言特征对小型语言模型进行微调，以执行CoT推理。关键挑战在于，这些语言模型容易生成误导答案推理的幻觉推理链。为缓解此类错误的影响，我们提出多模态CoT（Multimodal-CoT），在解耦训练框架中融入视觉特征。该框架将理由生成与答案推断分为两个阶段。通过在两个阶段均引入视觉特征，模型能够生成有效促进答案推断的理由链。采用Multimodal-CoT后，我们参数量低于10亿的模型在ScienceQA基准测试中相比先前最先进的LLM（GPT-3.5）提升16%（75.17%→91.68%），甚至超越人类表现。代码已开源：https://github.com/amazon-science/mm-cot。