Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. Traditional approaches often depend on meticulously designed prompts, high-quality examples, or additional reward models for in-context learning, supervised fine-tuning, or RLHF. Reinforcement learning (RL) presents a dynamic alternative for LLMs to overcome these dependencies by engaging directly with task-specific environments. Nonetheless, it faces significant hurdles: 1) instability stemming from the exponentially vast action space requiring exploration; 2) challenges in assigning token-level credit based on action-level reward signals, resulting in discord between maximizing rewards and accurately modeling corpus data. In response to these challenges, we introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. At the heart of ETPO is our novel per-token soft Bellman update, designed to harmonize the RL process with the principles of language modeling. This methodology decomposes the Q-function update from a coarse action-level view to a more granular token-level perspective, backed by theoretical proof of optimization consistency. Crucially, this decomposition renders linear time complexity in action exploration. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks; results underline ETPO's potential as a robust method for refining the interactive decision-making capabilities of language agents. For a more detailed preliminary work describing our motivation for token-level decomposition and applying it in PPO methods, please refer to arXiv:2405.15821.

翻译：大型语言模型（LLM）在交互式决策任务中展现出作为智能体的潜力。传统方法通常依赖于精心设计的提示、高质量示例或额外的奖励模型，以进行上下文学习、监督微调或RLHF。强化学习（RL）为LLM提供了一种动态替代方案，使其能够通过与任务特定环境直接交互来克服这些依赖。然而，该方法面临显著障碍：1）因需探索指数级庞大的动作空间而导致的不稳定性；2）基于动作级奖励信号分配令牌级信用的挑战，导致奖励最大化与语料数据精确建模之间的不协调。针对这些挑战，我们提出熵正则化令牌级策略优化（ETPO），这是一种专为令牌级优化LLM设计的熵增强强化学习方法。ETPO的核心是我们新颖的每令牌软贝尔曼更新，旨在使强化学习过程与语言建模原则相协调。该方法将Q函数更新从粗粒度的动作级视角分解为更细粒度的令牌级视角，并辅以优化一致性的理论证明。关键的是，这种分解使得动作探索具有线性时间复杂度。我们在模拟环境中评估ETPO的有效性，该环境将数据科学代码生成建模为一系列多步交互任务；结果凸显了ETPO作为提升语言智能体交互决策能力的稳健方法的潜力。关于我们令牌级分解动机及其在PPO方法中应用的更详细前期工作，请参阅arXiv:2405.15821。