The ability to predict upcoming events has been hypothesized to comprise a key aspect of natural and machine cognition. This is supported by trends in deep reinforcement learning (RL), where self-supervised auxiliary objectives such as prediction are widely used to support representation learning and improve task performance. Here, we study the effects predictive auxiliary objectives have on representation learning across different modules of an RL system and how these mimic representational changes observed in the brain. We find that predictive objectives improve and stabilize learning particularly in resource-limited architectures, and we identify settings where longer predictive horizons better support representational transfer. Furthermore, we find that representational changes in this RL system bear a striking resemblance to changes in neural activity observed in the brain across various experiments. Specifically, we draw a connection between the auxiliary predictive model of the RL system and hippocampus, an area thought to learn a predictive model to support memory-guided behavior. We also connect the encoder network and the value learning network of the RL system to visual cortex and striatum in the brain, respectively. This work demonstrates how representation learning in deep RL systems can provide an interpretable framework for modeling multi-region interactions in the brain. The deep RL perspective taken here also suggests an additional role of the hippocampus in the brain -- that of an auxiliary learning system that benefits representation learning in other regions.

翻译：预测即将发生事件的能力被认为是自然认知与机器认知的关键方面。这一观点在深度强化学习的发展趋势中得到印证——诸如预测等自监督辅助目标被广泛用于支持表征学习并提升任务性能。本研究考察了预测性辅助目标对强化学习系统不同模块表征学习的影响，以及这些机制如何模仿大脑中观察到的表征变化。我们发现，预测目标能有效改善并稳定学习过程，特别是在资源受限架构中，同时识别出更长预测周期更有利于表征迁移的特定场景。此外，我们观察到该强化学习系统中的表征变化与脑内多种实验观测到的神经活动变化存在显著相似性。具体而言，我们将强化学习系统的辅助预测模型与海马体建立联系——该脑区被认为通过构建预测模型支持记忆引导行为。同时将编码器网络和价值学习网络分别与视觉皮层和纹状体建立对应关系。本研究表明，深度强化学习系统的表征学习可为建模脑内多区域交互提供可解释框架。本文采用的深度强化学习视角还揭示了海马体在大脑中的另一潜在功能——作为辅助学习系统促进其他脑区的表征学习。