Covering skill (a.k.a., option) discovery has been developed to improve the exploration of RL in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. Given that joint state space grows exponentially with the number of agents in multi-agent systems, existing researches still relying on single-agent skill discovery either become prohibitive or fail to directly discover joint skills that improve the connectivity of the joint state space. In this paper, we propose multi-agent skill discovery which enables the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector using the Laplacian spectrum of individual agents' transition graphs. Further, considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method by estimating eigenfunctions through NN-based representation learning techniques. The evaluation on multi-agent tasks built with simulators like Mujoco, shows that the proposed algorithm can successfully identify multi-agent skills, and significantly outperforms the state-of-the-art. Codes are available at: https://github.itap.purdue.edu/Clan-labs/Scalable_MAOD_via_KP.

翻译：覆盖技能（也称为选项发现）通过利用状态转移图Fiedler向量提供的嵌入空间连接最远状态，已被用于改进稀疏奖励信号下单智能体场景中的强化学习探索。由于多智能体系统中联合状态空间随智能体数量呈指数增长，现有仍依赖单智能体技能发现的研究要么变得难以处理，要么无法直接发现能改善联合状态空间连通性的联合技能。本文提出多智能体技能发现方法，可实现简便分解。核心思想是将联合状态空间近似为Kronecker图，据此可利用各智能体转移图的拉普拉斯谱直接估计其Fiedler向量。进一步针对无限规模状态空间任务中直接计算拉普拉斯谱的不可解性，提出基于神经网络表征学习技术估计特征函数的深度学习扩展方法。在基于Mujoco等模拟器构建的多智能体任务上进行的评估表明，所提算法能成功识别多智能体技能，并显著优于现有最优方法。代码见：https://github.itap.purdue.edu/Clan-labs/Scalable_MAOD_via_KP。