The successor representation (SR) provides a powerful framework for decoupling predictive dynamics from rewards, enabling rapid generalisation across reward configurations. However, the classical SR is limited by its inherent policy dependence: policies change due to ongoing learning, environmental non-stationarities, and changes in task demands, making established predictive representations obsolete. Furthermore, in topologically complex environments, SRs suffer from spectral diffusion, leading to dense and overlapping features that scale poorly. Here we propose the Hierarchical Successor Representation (HSR) for overcoming these limitations. By incorporating temporal abstractions into the construction of predictive representations, HSR learns stable state features which are robust to task-induced policy changes. Applying non-negative matrix factorisation (NMF) to the HSR yields a sparse, low-rank state representation that facilitates highly sample-efficient transfer to novel tasks in multi-compartmental environments. Further analysis reveals that HSR-NMF discovers interpretable topological structures, providing a policy-agnostic hierarchical map that effectively bridges model-free optimality and model-based flexibility. Beyond providing a useful basis for task-transfer, we show that HSR's temporally extended predictive structure can also be leveraged to drive efficient exploration, effectively scaling to large, procedurally generated environments.

翻译：后继表征（SR）为解耦预测动力学与奖励提供了一个强大的框架，能够实现跨奖励配置的快速泛化。然而，经典SR受限于其固有的策略依赖性：策略因持续学习、环境非平稳性及任务需求变化而改变，导致已建立的预测表征过时。此外，在拓扑复杂的环境中，SR会出现谱扩散现象，产生密度高、重叠大的特征，可扩展性差。为此，我们提出层级式后继表征（HSR）以克服这些局限。通过将时间抽象融入预测表征的构建过程，HSR能学习到对任务引发的策略变化具有鲁棒性的稳定状态特征。对HSR应用非负矩阵分解（NMF）可得到稀疏低秩的状态表征，从而在多隔间环境中实现样本高效地向新任务迁移。进一步分析表明，HSR-NMF能发现可解释的拓扑结构，提供一张与策略无关的层级地图，有效桥接了无模型最优性与基于模型灵活性。除为任务迁移提供有用基础外，我们还证明HSR的时间扩展预测结构也可用于驱动高效探索，有效扩展至大规模程序化生成的环境。