Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models.

翻译：从负样本中学习对于提升大语言模型的推理能力具有巨大潜力，然而现有方法将所有错误答案视为同等信息量，忽视了样本质量的关键作用。为此，我们提出合理负样本方法，该方法能合成高质量负样本，这些样本在保持预期格式与结构连贯性的同时最终产生错误答案。PNS通过逆向强化学习训练专用模型，其复合奖励函数综合了格式合规性、准确性反转、奖励模型评估及思维链评价，生成的响应与正确答案几乎无法区分。我们进一步在七个数学推理基准上对三种骨干模型验证了PNS作为即插即用偏好优化数据源的有效性。实验结果表明，PNS始终优于其他负样本合成方法，相较于强化学习训练模型平均提升2.03%。