Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. With Multimodal-CoT, our model under 1 billion parameters outperforms the previous state-of-the-art LLM (GPT-3.5) by 16% (75.17%->91.68%) on the ScienceQA benchmark and even surpasses human performance. Code is publicly available available at https://github.com/amazon-science/mm-cot.

翻译：大型语言模型（LLMs）通过利用链式推理（CoT）提示生成中间推理链作为推断答案的依据，在复杂推理任务中展现了卓越性能。然而，现有CoT研究主要聚焦于语言模态。我们提出多模态CoT（Multimodal-CoT），将语言（文本）与视觉（图像）模态纳入一个两阶段框架，该框架分别处理依据生成与答案推断。通过这种方式，答案推断能够利用基于多模态信息生成的更优依据。采用Multimodal-CoT方法，我们参数量不足10亿的模型在ScienceQA基准测试中比此前最先进的LLM（GPT-3.5）提升16%（75.17%→91.68%），甚至超越人类表现。代码已公开，访问地址为https://github.com/amazon-science/mm-cot。