Reinforcement learning (RL) has shown great promise for developing dialogue management (DM) agents that are non-myopic, conduct rich conversations, and maximize overall user satisfaction. Despite recent developments in RL and language models (LMs), using RL to power conversational chatbots remains challenging, in part because RL requires online exploration to learn effectively, whereas collecting novel human-bot interactions can be expensive and unsafe. This issue is exacerbated by the combinatorial action spaces facing these algorithms, as most LM agents generate responses at the word level. We develop a variety of RL algorithms, specialized to dialogue planning, that leverage recent Mixture-of-Expert Language Models (MoE-LMs) -- models that capture diverse semantics, generate utterances reflecting different intents, and are amenable for multi-turn DM. By exploiting MoE-LM structure, our methods significantly reduce the size of the action space and improve the efficacy of RL-based DM. We evaluate our methods in open-domain dialogue to demonstrate their effectiveness w.r.t.\ the diversity of intent in generated utterances and overall DM performance.

翻译：强化学习(RL)在开发非近视性、能够进行丰富对话并最大化用户整体满意度的对话管理(DM)智能体方面展现出巨大潜力。尽管RL和语言模型(LM)近年来取得了进展，但通过RL增强对话聊天机器人仍然颇具挑战，部分原因在于RL需要在线探索才能有效学习，而收集新的人机交互数据既昂贵又存在安全隐患。由于多数基于LM的智能体在词级别生成响应，其面临的组合动作空间进一步加剧了该问题。我们开发了一系列专门针对对话规划的RL算法，这些算法利用最新的混合专家语言模型(MoE-LM)——此类模型能够捕获多样语义、生成反映不同意图的语句，并适用于多轮对话管理。通过利用MoE-LM的结构特性，我们提出的方法显著降低了动作空间的规模，并提升了基于RL的对话管理效能。我们在开放域对话中评估了这些方法，验证了其在生成语句的意图多样性及整体对话管理性能方面的有效性。