Reinforcement learning (RL) has shown great promise for developing dialogue management (DM) agents that are non-myopic, conduct rich conversations, and maximize overall user satisfaction. Despite recent developments in RL and language models (LMs), using RL to power conversational chatbots remains challenging, in part because RL requires online exploration to learn effectively, whereas collecting novel human-bot interactions can be expensive and unsafe. This issue is exacerbated by the combinatorial action spaces facing these algorithms, as most LM agents generate responses at the word level. We develop a variety of RL algorithms, specialized to dialogue planning, that leverage recent Mixture-of-Expert Language Models (MoE-LMs) -- models that capture diverse semantics, generate utterances reflecting different intents, and are amenable for multi-turn DM. By exploiting MoE-LM structure, our methods significantly reduce the size of the action space and improve the efficacy of RL-based DM. We evaluate our methods in open-domain dialogue to demonstrate their effectiveness w.r.t.\ the diversity of intent in generated utterances and overall DM performance.

翻译：强化学习（RL）在开发非短视型对话管理（DM）代理方面展现出巨大潜力，这类代理能够进行丰富的对话交互，并最大化用户整体满意度。尽管强化学习与语言模型（LM）近年来取得显著进展，但利用强化学习驱动对话聊天机器人仍面临挑战，部分原因在于强化学习需要通过在线探索实现有效学习，而收集新的人机交互数据既昂贵又存在安全隐患。这一问题因算法所需应对的组合动作空间而进一步加剧——多数语言模型代理在词级层面生成响应。我们开发了多种专用于对话规划的强化学习算法，这些算法利用了最新的混合专家语言模型（MoE-LMs）——该类模型能够捕获多样化语义、生成反映不同意图的语句，并适用于多轮对话管理。通过利用MoE-LM的结构特性，我们的方法显著缩减了动作空间规模，提升了基于强化学习的对话管理效能。我们在开放域对话场景中评估了所提方法，验证了其在生成语句的意图多样性及整体对话管理性能方面的有效性。