Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline m ulti-agent RL algorithm with implicit global-to-local v alue regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.

翻译：离线强化学习（RL）近年来因其能够从离线数据集中学习策略而无需与环境交互的出色能力而备受关注。尽管在单智能体场景下取得了一些成功，离线多智能体强化学习（MARL）仍然是一个挑战。巨大的联合状态-动作空间和耦合的多智能体行为为离线策略优化带来了额外的复杂性。大多数现有离线MARL研究仅对单个智能体应用与离线数据相关的正则化，而未在全局层面充分考虑多智能体系统。本文提出OMIGA，一种新的离线多智能体RL算法，其具备隐式全局到局部价值正则化能力。OMIGA提供了一个原则性框架，将全局层面价值正则化转化为等价的隐式局部价值正则化，同时实现了样本内学习，从而巧妙地将多智能体价值分解与策略学习同离线正则化相结合。基于离线多智能体MuJoCo和星际争霸II微操任务的全面实验，我们展示了OMIGA在几乎所有任务中均取得了优于最先进离线MARL方法的卓越性能。