In recent years, reinforcement learning and its multi-agent analogue have achieved great success in solving various complex control problems. However, multi-agent reinforcement learning remains challenging both in its theoretical analysis and empirical design of algorithms, especially for large swarms of embodied robotic agents where a definitive toolchain remains part of active research. We use emerging state-of-the-art mean-field control techniques in order to convert many-agent swarm control into more classical single-agent control of distributions. This allows profiting from advances in single-agent reinforcement learning at the cost of assuming weak interaction between agents. However, the mean-field model is violated by the nature of real systems with embodied, physically colliding agents. Thus, we combine collision avoidance and learning of mean-field control into a unified framework for tractably designing intelligent robotic swarm behavior. On the theoretical side, we provide novel approximation guarantees for general mean-field control both in continuous spaces and with collision avoidance. On the practical side, we show that our approach outperforms multi-agent reinforcement learning and allows for decentralized open-loop application while avoiding collisions, both in simulation and real UAV swarms. Overall, we propose a framework for the design of swarm behavior that is both mathematically well-founded and practically useful, enabling the solution of otherwise intractable swarm problems.

翻译：近年来，强化学习及其多智能体变体在解决各类复杂控制问题方面取得了巨大成功。然而，多智能体强化学习在理论分析和算法经验设计上仍面临挑战，尤其是针对具有具身特征的机器人大规模集群，其完整技术链条仍处于活跃研究阶段。我们采用新兴的最优平均场控制技术，将多智能体集群控制转化为更经典的单智能体分布控制问题。这种方法能以假设智能体间弱交互为代价，借助单智能体强化学习的最新进展。但具身物理碰撞智能体的真实系统特性会违背平均场模型假设。因此，我们提出将碰撞避免与平均场控制学习统一融合的框架，以实现可处理的智能机器人集群行为设计。在理论层面，我们为连续空间和含碰撞避免的通用平均场控制提供了新型近似保证。在实践层面，我们证明该方法在仿真和真实无人机集群中均优于多智能体强化学习，且能在避免碰撞的同时实现去中心化开环控制。总体而言，我们提出了一个兼具数学严谨性与实际应用价值的集群行为设计框架，能够解决传统方法难以处理的集群控制问题。