LLMs are increasingly deployed as agents that interact with external environments and observe feedback such as execution results, error messages, and tool outputs. A well-functioning agent should be able to leverage this feedback to accurately assess its own performance. Yet we find a persistent reflection gap: LLM agents tend to mis-assess their own outputs after observing concrete environment feedback -- even for questions they correctly answered -- and standard RL barely helps due to a credit-assignment mismatch. To close this gap, we propose RefGRPO, a simple yet effective fix that augments standard RL algorithms with two key ingredients: a free calibration bonus computed by contrasting the agent's own reflection with the actual outcome (requiring no additional reward model, LLM judge, or external annotation), and a dynamic schedule on its coefficient. Compared to standard RL baselines, our method simultaneously improves reflection calibration (e.g., reduces underconfidence rate $44.4\% \to 7.7\%$) and task accuracy (e.g., $75.1\% \to 76.5\%$) on text-to-SQL across five benchmarks. The resulting calibrated reflection turns the agent into its own verifier grounded in environment feedback, which further enables (i) better self-improvement that uses reflections as pseudo-rewards without outcome supervision, and (ii) more effective test-time selective prediction by committing only to rollouts flagged as correct.

翻译：大型语言模型越来越多地被部署为与外部环境交互的智能体，并观察执行结果、错误消息和工具输出等反馈。一个运行良好的智能体应能利用这些反馈准确评估自身性能。然而，我们发现了持续的反思差距：在观察到具体环境反馈后，大型语言模型智能体倾向于错误评估自身输出——即使是对其正确回答的问题也是如此——而标准强化学习由于信用分配错配几乎无法解决这一问题。为弥补这一差距，我们提出RefGRPO，一种简单有效的修正方法，通过两个关键要素增强标准强化学习算法：一个免费校准增益（通过对比智能体自身反思与实际结果计算得出，无需额外奖励模型、大型语言模型评判或外部标注），以及其系数的动态调度。与标准强化学习基线相比，我们的方法在五个基准的文本到SQL任务中同时提升了反思校准（例如，将低估自信率从44.4%降至7.7%）和任务准确率（例如，从75.1%提升至76.5%）。由此产生的校准反思将智能体转变为基于环境反馈的自我验证器，从而进一步实现：（i）更优的自我改进，可在无结果监督的情况下将反思用作伪奖励；（ii）更有效的测试时选择性预测，仅对被标记为正确的执行结果进行提交。