Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open-ended environments with multi-modal observations, such as the popular dungeon-crawler game, NetHack. Intriguingly, the NeurIPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. In this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration of reinforcement learning with imitation learning. Our investigations produce a state-of-the-art neural agent that surpasses previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, we also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players.

翻译：神经策略学习方法在各类控制问题中取得了显著成果，涵盖从Atari游戏到模拟运动等领域。然而，这些方法在长期任务中表现不佳，尤其是在具有多模态观测的开放式环境（如经典地牢探险游戏NetHack）中。值得注意的是，NeurIPS 2021 NetHack挑战赛显示，符号智能体在游戏得分中位数上超越神经方法四倍以上。本文深入探究这一性能差距的成因，并对NetHack的神经策略学习展开系统性研究。为此，我们分析了获胜的符号智能体，通过扩展其代码库追踪内部策略选择过程，生成了目前规模最大的演示数据集之一。基于该数据集，我们研究了以下方面：(i) 动作层次结构的优势；(ii) 神经架构的优化；以及 (iii) 强化学习与模仿学习的融合。实验表明，我们所提出的最优神经智能体在离线设置下游戏得分中位数提升127%，在线设置下提升25%，全面超越此前所有纯神经策略模型。但我们也证实，单纯扩大模型规模仍不足以弥合与最优符号模型乃至顶尖人类玩家之间的性能差距。