Recent months have seen the emergence of a powerful new trend in which large language models (LLMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, are not optimized using environment-specific rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that are compatible with gradient-based learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and proposing action plans. Experimental results on various tasks demonstrate that the language agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment. This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.

翻译：近几个月来，出现了一股强大的新趋势，即大型语言模型被增强为自主语言代理，能够自行执行面向目标的多步骤任务，而不仅仅是回应用户的查询。然而，现有的大多数语言代理并未利用环境特定奖励进行优化。尽管部分代理通过口头反馈实现迭代改进，但其推理与规划方式与基于奖励的梯度学习不兼容。本文提出了一种基于原则的框架，通过学习回顾模型来强化大型语言代理——该模型利用策略梯度从环境反馈中自动调整语言代理提示。具体而言，我们提出的代理架构通过跨多个环境和任务学习奖励，微调预训练语言模型，该模型通过总结先前失败尝试的根源并提出行动计划来优化语言代理提示。在多种任务上的实验结果表明，语言代理随时间持续改进，我们的方法显著优于未充分利用环境梯度的基线方法。这证明使用策略梯度优化来改进语言代理（我们相信本工作在此领域属首创之一）具有广阔前景，并可用于优化代理架构中的其他模型，从而持续提升代理性能。