DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to activate complex reasoning capabilities such as questioning and reflection in MLLMs, due to the absence of substantial high-quality multimodal reasoning data. To address this issue, we propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Specifically, we first construct a high-quality multimodal CoT dataset without human annotations by leveraging an existing MLLM and DeepSeek-R1 through modality bridging and data filtering to obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as cold-start initialization data for Vision-R1. To mitigate the optimization challenges caused by overthinking after cold start, we propose Progressive Thinking Suppression Training (PTST) strategy and employ Group Relative Policy Optimization (GRPO) with the hard formatting result reward function to gradually refine the model's ability to learn correct and complex reasoning processes on a 10K multimodal math dataset. Comprehensive experiments show our model achieves an average improvement of $\sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1. Scaling up the amount of multimodal math data in the RL training, Vision-R1-32B and Vison-R1-72B achieves 76.4% and 78.2% MathVista benchmark scores, respectively. The datasets and code will be released in: https://github.com/Osilly/Vision-R1 .

翻译：DeepSeek-R1-Zero 已成功证明，仅通过强化学习即可在大型语言模型中涌现出推理能力。受此突破启发，我们探索如何利用强化学习来增强多模态大语言模型的推理能力。然而，由于缺乏大量高质量的多模态推理数据，直接使用强化学习进行训练难以激活多模态大语言模型中的复杂推理能力，例如质疑与反思。为解决此问题，我们提出了推理型多模态大语言模型 Vision-R1，以提升多模态推理能力。具体而言，我们首先通过利用现有的多模态大语言模型和 DeepSeek-R1，借助模态桥接与数据过滤技术，构建了一个无需人工标注的高质量多模态思维链数据集——Vision-R1-cold 数据集，其规模为 20 万条。该数据集作为 Vision-R1 的冷启动初始化数据。为了缓解冷启动后因过度思考带来的优化挑战，我们提出了渐进式思维抑制训练策略，并采用组相对策略优化算法配合基于硬格式化结果的奖励函数，在一个包含 1 万条多模态数学题的数据集上逐步精炼模型学习正确且复杂推理过程的能力。全面的实验表明，我们的模型在多个多模态数学推理基准测试中平均提升了约 6%。Vision-R1-7B 在广泛使用的 MathVista 基准测试上取得了 73.5% 的准确率，仅比领先的推理模型 OpenAI O1 低 0.4%。通过扩大强化学习训练中多模态数学数据的规模，Vision-R1-32B 和 Vision-R1-72B 在 MathVista 基准测试上分别取得了 76.4% 和 78.2% 的分数。数据集和代码将在以下地址发布：https://github.com/Osilly/Vision-R1 。