The discovery of individual objectives in collective behavior of complex dynamical systems such as fish schools and bacteria colonies is a long-standing challenge. Inverse reinforcement learning is a potent approach for addressing this challenge but its applicability to dynamical systems, involving continuous state-action spaces and multiple interacting agents, has been limited. In this study, we tackle this challenge by introducing an off-policy inverse multi-agent reinforcement learning algorithm (IMARL). Our approach combines the ReF-ER techniques with guided cost learning. By leveraging demonstrations, our algorithm automatically uncovers the reward function and learns an effective policy for the agents. Through extensive experimentation, we demonstrate that the proposed policy captures the behavior observed in the provided data, and achieves promising results across problem domains including single agent models in the OpenAI gym and multi-agent models of schooling behavior. The present study shows that the proposed IMARL algorithm is a significant step towards understanding collective dynamics from the perspective of its constituents, and showcases its value as a tool for studying complex physical systems exhibiting collective behaviour.

翻译：复杂动态系统（如鱼群和细菌群落）中集体行为所蕴含的个体目标发现是一个长期存在的挑战。逆强化学习是应对这一挑战的有效方法，但其在涉及连续状态-动作空间及多智能体交互的动态系统中的应用仍十分有限。本研究通过引入一种离策略逆多智能体强化学习算法（IMARL）来攻克这一难题。我们的方法将ReF-ER技术与引导代价学习相结合。通过利用示范数据，该算法能自动揭示奖励函数，并为智能体学习到有效策略。大量实验表明，我们提出的策略能够捕捉到给定数据中呈现的行为，并在包括OpenAI gym中的单智能体模型以及集群行为多智能体模型在内的多个问题领域取得了令人满意的结果。本研究证实，所提出的IMARL算法是从个体角度理解集体动力学的重要进展，并展示了其作为研究呈现集体行为的复杂物理系统工具的价值。