The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1 model with a focus on coding tasks. It integrates reinforcement learning (RL) and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking capabilities. The framework includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and iteratively fine-tuning the policy model to initially produce pseudocode, followed by the generation of the full code. The report also addresses the opportunities and challenges in deploying o1-like models in real-world applications, suggesting transitioning to the System-2 paradigm and highlighting the imperative for environment state updates. Updated model progress and experimental results will be reported in subsequent versions. All source code, curated datasets, as well as the derived models will be disclosed at https://github.com/ADaM-BJTU/O1-CODER .

翻译：本技术报告介绍了O1-CODER，这是对OpenAI o1模型在代码任务上的复现尝试。该模型整合了强化学习（RL）与蒙特卡洛树搜索（MCTS），以增强模型的系统二思维能力。框架包含训练用于标准化代码测试的测试用例生成器（TCG）、使用MCTS生成带推理过程的代码数据，以及迭代微调策略模型以首先生成伪代码，继而生成完整代码。报告还探讨了在实际应用中部署类o1模型所面临的机遇与挑战，建议向系统二范式过渡，并强调了环境状态更新的必要性。更新后的模型进展与实验结果将在后续版本中报告。所有源代码、精选数据集及衍生模型均发布于https://github.com/ADaM-BJTU/O1-CODER。