Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. With Multimodal-CoT, our model under 1 billion parameters outperforms the previous state-of-the-art LLM (GPT-3.5) by 16 percentage points (75.17%->91.68% accuracy) on the ScienceQA benchmark and even surpasses human performance. Code is publicly available available at https://github.com/amazon-science/mm-cot.

翻译：大规模语言模型（LLMs）通过利用思维链（CoT）提示生成中间推理链作为推断答案的依据，在复杂推理任务上展现了卓越性能。然而，现有CoT研究主要聚焦于语言模态。我们提出多模态思维链（Multimodal-CoT）方法，将语言（文本）与视觉（图像）模态融入两阶段框架，分别进行理由生成与答案推断。通过这种方式，答案推断能够利用基于多模态信息生成的更优理由。采用Multimodal-CoT的模型参数量不足10亿，却在ScienceQA基准测试中比此前最先进的大语言模型（GPT-3.5）准确率提升16个百分点（从75.17%提升至91.68%），甚至超越人类表现。代码已开源，可在 https://github.com/amazon-science/mm-cot 获取。