A safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers. We propose a novel Reinforcement Learning approach that allows to directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to factual questions. Our method optimizes a reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidence. This encourages the model to align its confidence estimates with the actual predictive accuracy. The optimal policy under our reward design would result in perfectly calibrated confidence expressions. Unlike prior approaches that decouple confidence estimation from response generation, our method integrates confidence calibration seamlessly into the generative process of the LLM. Empirically, we demonstrate that models trained with our approach exhibit substantially improved calibration and generalize to unseen tasks without further fine-tuning, suggesting the emergence of general confidence awareness.

翻译：安全可信地使用大语言模型（LLMs）需要对其答案的置信度进行准确表达。我们提出了一种新颖的强化学习方法，可直接微调LLMs，使其在回答事实性问题时同步输出经过校准的置信度估计。该方法基于对数评分规则优化奖励函数，明确惩罚过度自信与自信不足两种情况，从而激励模型使其置信度估计与实际预测准确率保持一致。在我们的奖励设计下，最优策略将产生完全校准的置信度表达。与以往将置信度估计与响应生成解耦的方法不同，我们的方法将置信度校准无缝集成到LLM的生成过程中。实证研究表明，通过我们的方法训练的模型展现出显著改善的校准性能，并且无需进一步微调即可泛化至未见任务，这表明模型形成了通用的置信度感知能力。