Current techniques for post-training Large Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check.

翻译：当前大语言模型（LLM）的后训练技术依赖于昂贵的人工监督或外部验证器来提升数学推理和代码生成等任务的性能。然而，随着LLM解决问题能力的提升，任何进一步的改进都可能需要人类难以提供的高质量难题解决方案。因此，从无标注数据中学习在研究界正变得越来越具有吸引力。现有方法通过多数投票或将模型内部置信度转化为奖励，从模型的一致性中提取学习信号。尽管熵或自我确定性等内部一致性度量无需人工干预，但本研究表明，这些信号在大规模长期训练中并不可靠。为解决此问题，我们提出PRISM——一个统一的训练框架，该框架在缺乏真实标签的情况下，利用过程奖励模型（PRM）结合模型内部置信度来指导学习。我们证明，将PRM与自我确定性有效结合不仅能实现稳定的训练和更优的测试性能，还能有效约束模型的内部置信度。