Large language models (LLMs) demonstrate impressive reasoning abilities, but translating reasoning into actions in the real world remains challenging. In particular, it remains unclear how to complete a given task provably within a minimum number of interactions with the external environment, e.g., through an internal mechanism of reasoning. To this end, we propose a principled framework with provable regret guarantees to orchestrate reasoning and acting, which we call ``reason for future, act for now" (\texttt{RAFA}). Specifically, we design a prompt template for reasoning that learns from the memory buffer and plans a future trajectory over a long horizon (``reason for future"). At each step, the LLM agent takes the initial action of the planned trajectory (``act for now"), stores the collected feedback in the memory buffer, and reinvokes the reasoning routine to replan the future trajectory from the new state. The key idea is to cast reasoning in LLMs as learning and planning in Bayesian adaptive Markov decision processes (MDPs). Correspondingly, we prompt LLMs to form an updated posterior of the unknown environment from the memory buffer (learning) and generate an optimal trajectory for multiple future steps that maximizes a value function (planning). The learning and planning subroutines are performed in an "in-context" manner to emulate the actor-critic update for MDPs. Our theoretical analysis proves that the novel combination of long-term reasoning and short-term acting achieves a $\sqrt{T}$ regret. In particular, the regret bound highlights an intriguing interplay between the prior knowledge obtained through pretraining and the uncertainty reduction achieved by reasoning and acting. Our empirical validation shows that it outperforms various existing frameworks and achieves nearly perfect scores on a few benchmarks.

翻译：大型语言模型（LLMs）展现出令人印象深刻的推理能力，但将推理转化为现实世界中的行动仍具挑战性。特别是，如何通过内部推理机制，在与外部环境的最小交互次数内可证明地完成给定任务仍不明确。为此，我们提出了一个具有可证明遗憾保证的规范框架来协调推理与行动，称之为“为未来推理，为当下行动”（\texttt{RAFA}）。具体而言，我们设计了一个推理提示模板，该模板从记忆缓冲区中学习，并规划长期未来轨迹（“为未来推理”）。在每个步骤中，LLM代理执行规划轨迹的初始动作（“为当下行动”），将收集到的反馈存储在记忆缓冲区中，并重新调用推理程序从新状态重新规划未来轨迹。关键思想是将LLM中的推理建模为贝叶斯自适应马尔可夫决策过程（MDPs）中的学习与规划。相应地，我们引导LLM从记忆缓冲区中形成未知环境的更新后验（学习），并生成最大化价值函数的多步最优未来轨迹（规划）。学习与规划子程序以“上下文内”方式执行，以模拟MDPs的演员-评论家更新。我们的理论分析证明，长期推理与短期行动的新颖组合实现了$\sqrt{T}$的遗憾。特别地，遗憾界突显了预训练获得的先验知识与通过推理和行动实现的不确定性减少之间的有趣相互作用。我们的实证验证表明，该框架优于多种现有框架，并在若干基准测试中实现了近乎完美的分数。