We consider the problem of cooperative exploration where multiple robots need to cooperatively explore an unknown region as fast as possible. Multi-agent reinforcement learning (MARL) has recently become a trending paradigm for solving this challenge. However, existing MARL-based methods adopt action-making steps as the metric for exploration efficiency by assuming all the agents are acting in a fully synchronous manner: i.e., every single agent produces an action simultaneously and every single action is executed instantaneously at each time step. Despite its mathematical simplicity, such a synchronous MARL formulation can be problematic for real-world robotic applications. It can be typical that different robots may take slightly different wall-clock times to accomplish an atomic action or even periodically get lost due to hardware issues. Simply waiting for every robot being ready for the next action can be particularly time-inefficient. Therefore, we propose an asynchronous MARL solution, Asynchronous Coordination Explorer (ACE), to tackle this real-world challenge. We first extend a classical MARL algorithm, multi-agent PPO (MAPPO), to the asynchronous setting and additionally apply action-delay randomization to enforce the learned policy to generalize better to varying action delays in the real world. Moreover, each navigation agent is represented as a team-size-invariant CNN-based policy, which greatly benefits real-robot deployment by handling possible robot lost and allows bandwidth-efficient intra-agent communication through low-dimensional CNN features. We first validate our approach in a grid-based scenario. Both simulation and real-robot results show that ACE reduces over 10% actual exploration time compared with classical approaches. We also apply our framework to a high-fidelity visual-based environment, Habitat, achieving 28% improvement in exploration efficiency.

翻译：我们考虑多机器人协同探索问题，即多个机器人需尽可能快地协同探索未知区域。近年来，多智能体强化学习（MARL）已成为解决该挑战的热门范式。然而，现有基于MARL的方法将动作执行步数作为探索效率的度量标准，其前提是所有智能体完全同步运行：即在每个时间步，每个智能体同时生成动作并瞬时执行。尽管这种同步MARL框架在数学上简洁，但在实际机器人应用中存在问题。不同机器人执行单个原子动作的壁钟时间可能略有差异，甚至因硬件故障周期性丢失。若每步都等待所有机器人就绪，将导致时间效率低下。为此，我们提出异步MARL解决方案——异步协调探索器（ACE）以应对这一实际挑战。首先，我们将经典MARL算法——多智能体PPO（MAPPO）扩展至异步场景，并引入动作延迟随机化技术，使学习策略能更好适应实际环境中变化的动作延迟。此外，每个导航智能体被表示为团队规模不变的CNN策略网络，既可通过处理可能的机器人丢失问题显著提升实体机器人部署鲁棒性，又通过低维CNN特征实现带宽高效的智能体间通信。我们首先在网格场景中验证该方法。仿真与实体实验均表明，与经典方法相比，ACE可将实际探索时间降低10%以上。我们还将框架应用于高保真视觉环境Habitat，探索效率提升达28%。