Reinforcement learning (RL) agents make decisions using nothing but observations from the environment, and consequently, heavily rely on the representations of those observations. Though some recent breakthroughs have used vector-based categorical representations of observations, often referred to as discrete representations, there is little work explicitly assessing the significance of such a choice. In this work, we provide a thorough empirical investigation of the advantages of representing observations as vectors of categorical values within the context of reinforcement learning. We perform evaluations on world-model learning, model-free RL, and ultimately continual RL problems, where the benefits best align with the needs of the problem setting. We find that, when compared to traditional continuous representations, world models learned over discrete representations accurately model more of the world with less capacity, and that agents trained with discrete representations learn better policies with less data. In the context of continual RL, these benefits translate into faster adapting agents. Additionally, our analysis suggests that the observed performance improvements can be attributed to the information contained within the latent vectors and potentially the encoding of the discrete representation itself.

翻译：强化学习（RL）智能体仅通过环境观测做出决策，因此高度依赖这些观测的表征方式。尽管近期部分突破性工作采用基于向量的分类表征（常称为离散表征），但鲜有研究系统评估这种选择的重要性。本研究在强化学习背景下，对观测值以分类值向量形式表征的优势进行了全面的实证探究。我们在世界模型学习、无模型强化学习以及最终在持续强化学习问题中开展评估——这些场景下离散表征的优势与问题设置的需求最为契合。研究发现：与传统连续表征相比，基于离散表征学习的世界模型能以更少容量更准确地建模世界；采用离散表征训练的智能体能用更少数据学习到更优策略。在持续强化学习场景中，这些优势转化为更快的智能体适应能力。此外，我们的分析表明，可观测的性能提升可归因于潜向量所包含的信息，以及离散表征本身的编码机制。