In this paper we investigate the properties of representations learned by deep reinforcement learning systems. Much of the early work on representations for reinforcement learning focused on designing fixed-basis architectures to achieve properties thought to be desirable, such as orthogonality and sparsity. In contrast, the idea behind deep reinforcement learning methods is that the agent designer should not encode representational properties, but rather that the data stream should determine the properties of the representation -- good representations emerge under appropriate training schemes. In this paper we bring these two perspectives together, empirically investigating the properties of representations that support transfer in reinforcement learning. We introduce and measure six representational properties over more than 25 thousand agent-task settings. We consider Deep Q-learning agents with different auxiliary losses in a pixel-based navigation environment, with source and transfer tasks corresponding to different goal locations. We develop a method to better understand why some representations work better for transfer, through a systematic approach varying task similarity and measuring and correlating representation properties with transfer performance. We demonstrate the generality of the methodology by investigating representations learned by a Rainbow agent that successfully transfer across games modes in Atari 2600.

翻译：本文研究了深度强化学习系统所学表征的性质。早期关于强化学习表征的研究主要聚焦于设计固定基架构，以实现正交性和稀疏性等被认为可取的性质。相比之下，深度强化学习方法的核心思想是：智能体设计者不应预先编码表征性质，而应由数据流决定表征的属性——适当的训练方案能够催生出优质表征。本文将这两种视角相结合，通过实验探究了支持迁移学习的强化学习表征性质。我们引入并测量了六项表征性质，涉及超过2.5万个智能体-任务设置。在基于像素的导航环境中，我们考察了采用不同辅助损失的深度Q学习智能体，其中源任务与迁移任务对应于不同的目标位置。我们开发了一种方法，通过系统性地变化任务相似度，测量表征性质并与迁移性能进行相关性分析，从而更深入地理解为何某些表征更有利于迁移。通过在Atari 2600游戏模式间成功迁移的Rainbow智能体所习得表征的研究，我们验证了该方法的普适性。