DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to activate complex reasoning capabilities such as questioning and reflection in MLLMs, due to the absence of substantial high-quality multimodal reasoning data. To address this issue, we propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Specifically, we first construct a high-quality multimodal CoT dataset without human annotations by leveraging an existing MLLM and DeepSeek-R1 through modality bridging and data filtering to obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as cold-start initialization data for Vision-R1. To mitigate the optimization challenges caused by overthinking after cold start, we propose Progressive Thinking Suppression Training (PTST) strategy and employ Group Relative Policy Optimization (GRPO) with the hard formatting result reward function to gradually refine the model's ability to learn correct and complex reasoning processes on a 10K multimodal math dataset. Comprehensive experiments show our model achieves an average improvement of $\sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1. Scaling up the amount of multimodal math data in the RL training, Vision-R1-32B and Vison-R1-72B achieves 76.4% and 78.2% MathVista benchmark scores, respectively. The datasets and code will be released in: https://github.com/Osilly/Vision-R1 .

翻译：DeepSeek-R1-Zero 已成功证明，仅通过强化学习（RL）即可在大语言模型（LLMs）中涌现出推理能力。受此突破启发，我们探索如何利用 RL 来增强多模态大语言模型（MLLMs）的推理能力。然而，由于缺乏大量高质量的多模态推理数据，直接使用 RL 进行训练难以激活 MLLMs 中诸如提问和反思等复杂推理能力。为解决此问题，我们提出了推理型多模态大语言模型 Vision-R1，以提升多模态推理能力。具体而言，我们首先通过利用现有的 MLLM 和 DeepSeek-R1，借助模态桥接和数据过滤技术，在没有人工标注的情况下构建了一个高质量的多模态思维链（CoT）数据集，即包含 20 万条数据的 Vision-R1-cold 数据集。它作为 Vision-R1 的冷启动初始化数据。为了缓解冷启动后因过度思考带来的优化挑战，我们提出了渐进式思维抑制训练（PTST）策略，并采用分组相对策略优化（GRPO）配合硬格式化结果奖励函数，在一个包含 1 万条数据的多模态数学数据集上逐步精炼模型学习正确且复杂推理过程的能力。全面的实验表明，我们的模型在各种多模态数学推理基准测试中平均提升了约 6%。Vision-R1-7B 在广泛使用的 MathVista 基准测试中达到了 73.5% 的准确率，仅比领先的推理模型 OpenAI O1 低 0.4%。通过扩大 RL 训练中多模态数学数据的规模，Vision-R1-32B 和 Vision-R1-72B 分别在 MathVista 基准测试中取得了 76.4% 和 78.2% 的分数。数据集和代码将在以下地址发布：https://github.com/Osilly/Vision-R1。