Covering skill (a.k.a., option) discovery has been developed to improve the exploration of reinforcement learning in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. However, these option discovery methods cannot be directly extended to multi-agent scenarios, since the joint state space grows exponentially with the number of agents in the system. Thus, existing researches on adopting options in multi-agent scenarios still rely on single-agent option discovery and fail to directly discover the joint options that can improve the connectivity of the joint state space of agents. In this paper, we show that it is indeed possible to directly compute multi-agent options with collaborative exploratory behaviors among the agents, while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph -- the Kronecker product of individual agents' state transition graphs, based on which we can directly estimate the Fiedler vector of the joint state space using the Laplacian spectrum of individual agents' transition graphs. This decomposition enables us to efficiently construct multi-agent joint options by encouraging agents to connect the sub-goal joint states which are corresponding to the minimum or maximum values of the estimated joint Fiedler vector. The evaluation based on multi-agent collaborative tasks shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options, in terms of both faster exploration and higher cumulative rewards.

翻译：摘要：覆盖技能（亦称选项）发现通过连接状态转移图Fiedler向量所提供嵌入空间中的最远状态，已被开发用于改善单智能体场景下稀疏奖励信号强化学习的探索效率。然而，这些选项发现方法无法直接扩展到多智能体场景，因为联合状态空间随系统中智能体数量呈指数增长。因此，现有关于多智能体场景中采用选项的研究仍依赖单智能体选项发现，未能直接发现能改善智能体联合状态空间连通性的联合选项。本文证明，在保持分解便利性的同时，直接计算具有智能体间协作探索行为的多智能体选项确实可行。核心思想是将联合状态空间近似为克罗内克图——各智能体状态转移图的克罗内克积，基于此可利用单智能体转移图的拉普拉斯谱直接估计联合状态空间的Fiedler向量。这种分解使我们能够通过激励智能体连接与估计联合Fiedler向量最小值或最大值对应的子目标联合状态，高效构建多智能体联合选项。基于多智能体协作任务的评估表明，所提算法能成功识别多智能体选项，且在探索速度和累积奖励两方面均显著优于使用单智能体选项或不使用选项的现有方法。