Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

翻译：大型语言模型（LLMs）日益被用作与外部环境（如游戏、编译器、API）交互的目标驱动型智能体。然而，这些语言智能体如何快速高效地从试错中学习仍面临挑战，因为传统强化学习方法需要大量训练样本和昂贵的模型微调。我们提出反思（Reflexion）——一种新颖框架，其通过语言反馈而非权重更新来强化语言智能体。具体而言，反思智能体通过言语方式反思任务反馈信号，随后在情景记忆缓冲区中维护自身的反思性文本，以在后续试验中诱导更优决策。该框架具有高度灵活性，可整合多种类型（标量值或自由形式语言）和来源（外部或内部模拟）的反馈信号，并在跨不同任务（序列决策、编程、语言推理）的实验中较基线智能体取得显著提升。例如，在HumanEval编程基准测试中，反思框架实现了91%的pass@1准确率，超越了此前取得80%准确率的最先进GPT-4。我们还通过不同反馈信号、反馈整合方法及智能体类型的消融与对比分析研究，揭示了各因素对性能的影响机理。