Improving the reasoning abilities of large language models (LLMs) has largely relied on iterative self-training with model-generated data. While effective at boosting accuracy, existing approaches primarily reinforce successful reasoning paths, incurring a substantial calibration cost: models become overconfident and lose the ability to represent uncertainty. This failure has been characterized as a form of model collapse in alignment, where predictive distributions degenerate toward low-variance point estimates. We address this issue by reframing reasoning training as an epistemic learning problem, in which models must learn not only how to reason, but also when their reasoning should be trusted. We propose epistemically-calibrated reasoning (EpiCaR) as a training objective that jointly optimizes reasoning performance and calibration, and instantiate it within an iterative supervised fine-tuning framework using explicit self-evaluation signals. Experiments on Llama-3 and Qwen-3 families demonstrate that our approach achieves Pareto-superiority over standard baselines in both accuracy and calibration, particularly in models with sufficient reasoning capacity (e.g., 3B+). This framework generalizes effectively to OOD mathematical reasoning (GSM8K) and code generation (MBPP). Ultimately, our approach enables a 3X reduction in inference compute, matching the K=30 performance of STaR with only K=10 samples in capable models.

翻译：提升大型语言模型（LLMs）的推理能力在很大程度上依赖于利用模型生成的数据进行迭代式自训练。尽管现有方法能有效提高准确率，但它们主要强化成功的推理路径，从而产生了显著的校准代价：模型变得过度自信，并丧失了表征不确定性的能力。这种失败被描述为对齐过程中的一种模型坍缩形式，其中预测分布退化为低方差的点估计。我们通过将推理训练重新定义为认知学习问题来解决此问题，在该问题中，模型不仅需要学习如何进行推理，还需要学习何时应信任其推理。我们提出认知校准推理（EpiCaR）作为联合优化推理性能和校准的训练目标，并利用显式的自我评估信号，在迭代式监督微调框架中实例化该目标。在Llama-3和Qwen-3系列模型上的实验表明，我们的方法在准确率和校准方面均实现了对标准基线的帕累托优势，尤其是在具备足够推理能力的模型（例如3B+参数规模）中。该框架能有效泛化到分布外数学推理（GSM8K）和代码生成（MBPP）任务。最终，我们的方法能够在具备能力的模型中，仅使用K=10个样本即可匹配STaR方法K=30个样本的性能，从而实现推理计算量3倍的缩减。