Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. For example, a teacher might try to understand their student's current comprehension level to tailor their instruction accordingly, and a travel agent might ask questions of their customer to understand their preferences in order to recommend activities they might enjoy. LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they are not trained to optimize for overall conversational outcomes after multiple turns of interaction. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue. Our key insight is that, though LLMs might not effectively solve goal-directed dialogue tasks out of the box, they can provide useful data for solving such tasks by simulating suboptimal but human-like behaviors. Given a textual description of a goal-directed dialogue task, we leverage LLMs to sample diverse synthetic rollouts of hypothetical in-domain human-human interactions. Our algorithm then utilizes this dataset with offline reinforcement learning to train an interactive conversational agent that can optimize goal-directed objectives over multiple turns. In effect, the LLM produces examples of possible interactions, and RL then processes these examples to learn to perform more optimal interactions. Empirically, we show that our proposed approach achieves state-of-the-art performance in various goal-directed dialogue tasks that include teaching and preference elicitation.

翻译：大型语言模型（LLMs）已成为众多自然语言任务中强大且通用的解决方案。然而，语言生成最重要的应用往往是交互式的，即智能体需与人类对话以达成预期目标。例如，教师可能试图理解学生当前的理解水平来调整教学策略，旅行顾问则需询问客户偏好以推荐合适的活动。采用监督微调或标准RLHF中的"单步"强化学习训练的LLMs，在需要多轮交互中优化整体对话结果的目标导向任务上可能存在困难。本研究探索了一种基于强化学习适配LLMs以应对此类目标导向对话的新方法。我们的核心洞察在于：尽管LLMs难以直接有效解决目标导向对话任务，但通过模拟次优但类人的行为，它们能够为这类任务的解决提供有效数据。给定目标导向对话任务的文本描述，我们利用LLMs对假设的领域内人际交互进行多样化合成轨迹采样。该算法随后使用离线强化学习处理此数据集，训练一个能通过多轮交互优化目标导向目标的交互式对话智能体。本质上，LLMs生成可能的交互示例，RL则通过处理这些示例来学习执行更优的交互。实验表明，我们提出的方法在包括教学与偏好挖掘在内的多种目标导向对话任务中达到了当前最优性能。