Answering complex natural language questions often necessitates multi-step reasoning and integrating external information. Several systems have combined knowledge retrieval with a large language model (LLM) to answer such questions. These systems, however, suffer from various failure cases, and we cannot directly train them end-to-end to fix such failures, as interaction with external knowledge is non-differentiable. To address these deficiencies, we define a ReAct-style LLM agent with the ability to reason and act upon external knowledge. We further refine the agent through a ReST-like method that iteratively trains on previous trajectories, employing growing-batch reinforcement learning with AI feedback for continuous self-improvement and self-distillation. Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model that achieves comparable performance on challenging compositional question-answering benchmarks with two orders of magnitude fewer parameters.

翻译：回答复杂的自然语言问题通常需要多步推理并整合外部信息。多个系统已将知识检索与大语言模型相结合来回答此类问题。然而，这些系统会遭受各种失败情况，且由于与外部知识的交互不可微分，我们无法通过端到端训练直接修复这些缺陷。为解决这些问题，我们定义了一种能够基于外部知识进行推理和行动的ReAct风格大语言模型智能体。我们进一步通过类似ReST的方法改进该智能体——该方法借助AI反馈的递增批次强化学习，基于先前轨迹进行迭代训练，实现持续自我改进与自我蒸馏。从提示驱动的初始大型模型开始，仅需两次算法迭代，我们就能生成一个微调后的小型模型，该模型在具有挑战性的组合式问答基准测试中以少两个数量级的参数数量达到可比性能。