Reinforcement learning has become the central approach for language models (LMs) to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.

翻译：强化学习已成为语言模型从环境奖励或反馈中学习的核心方法。在实践中，环境反馈通常是稀疏且延迟的。从这类信号中学习具有挑战性，因为语言模型必须隐式推断观察到的失败应如何转化为未来迭代中的行为改变。我们提出了经验强化学习，这是一种将显式的经验-反思-巩固循环嵌入强化学习过程的训练范式。给定一个任务，模型生成初始尝试，接收环境反馈，并产生一个指导改进后第二次尝试的反思，其成功会被强化并内化到基础策略中。这一过程将反馈转化为结构化的行为修正，从而改善探索、稳定优化，同时在部署时无需额外推理成本即可保持收益。在稀疏奖励控制环境和智能体推理基准测试中，ERL 相较于强大的强化学习基线，持续提升了学习效率和最终性能，在复杂多步环境中实现了高达 +81% 的增益，在工具使用推理任务中实现了高达 +11% 的增益。这些结果表明，将显式自我反思整合到策略训练中，为将反馈转化为持久的行为改进提供了一种实用机制。