Intrinsically motivated exploration has proven useful for reinforcement learning, even without additional extrinsic rewards. When the environment is naturally represented as a graph, how to guide exploration best remains an open question. In this work, we propose a novel approach for exploring graph-structured data motivated by two theories of human curiosity: the information gap theory and the compression progress theory. The theories view curiosity as an intrinsic motivation to optimize for topological features of subgraphs induced by nodes visited in the environment. We use these proposed features as rewards for graph neural-network-based reinforcement learning. On multiple classes of synthetically generated graphs, we find that trained agents generalize to longer exploratory walks and larger environments than are seen during training. Our method computes more efficiently than the greedy evaluation of the relevant topological properties. The proposed intrinsic motivations bear particular relevance for recommender systems. We demonstrate that next-node recommendations considering curiosity are more predictive of human choices than PageRank centrality in several real-world graph environments, including MovieLens, Amazon Books, and Wikipedia.

翻译：内在动机驱动的探索已被证明对强化学习有效，即使在缺乏外部奖励的情况下也是如此。当环境天然以图结构呈现时，如何实现最优探索仍是一个未解问题。本研究提出一种受人类好奇心两种理论（信息差理论与压缩进展理论）启发的新型图结构数据探索方法。该理论将好奇心视为一种内在动机，旨在优化环境中所访问节点诱导的子图的拓扑特征。我们将这些特征作为基于图神经网络的强化学习的奖励信号。在多类合成生成的图上，我们发现训练后的智能体能够泛化至比训练时更长的探索路径及更大规模的环境。与相关拓扑属性的贪婪评估相比，我们的方法计算效率更高。所提出的内在动机对推荐系统具有特殊意义。实验表明，在MovieLens、Amazon Books及Wikipedia等多个真实图环境中，考虑好奇心的下一节点推荐比PageRank中心性更能预测人类选择。