Offline reinforcement learning (RL) that learns policies from offline datasets without environment interaction has received considerable attention in recent years. Compared with the rich literature in the single-agent case, offline multi-agent RL is still a relatively underexplored area. Most existing methods directly apply offline RL ingredients in the multi-agent setting without fully leveraging the decomposable problem structure, leading to less satisfactory performance in complex tasks. We present OMAC, a new offline multi-agent RL algorithm with coupled value factorization. OMAC adopts a coupled value factorization scheme that decomposes the global value function into local and shared components, and also maintains the credit assignment consistency between the state-value and Q-value functions. Moreover, OMAC performs in-sample learning on the decomposed local state-value functions, which implicitly conducts max-Q operation at the local level while avoiding distributional shift caused by evaluating out-of-distribution actions. Based on the comprehensive evaluations of the offline multi-agent StarCraft II micro-management tasks, we demonstrate the superior performance of OMAC over the state-of-the-art offline multi-agent RL methods.

翻译：离线强化学习（Reinforcement Learning, RL）通过从离线数据集中学习策略，无需与环境交互，近年来受到广泛关注。相较于单智能体场景下丰富的文献，离线多智能体RL仍是一个相对未充分探索的领域。现有方法大多直接将在离线RL组件应用于多智能体设置，未能充分利用问题的可分解结构，导致在复杂任务中性能欠佳。本文提出了一种新的离线多智能体RL算法——OMAC（Offline Multi-Agent RL with Coupled Value Factorization），该方法采用耦合值分解方案，将全局价值函数分解为局部和共享组件，同时保持状态-价值函数与Q-价值函数之间的信用分配一致性。此外，OMAC在分解后的局部状态-价值函数上进行样本内学习（in-sample learning），隐式地在局部层面实施max-Q操作，同时避免因评估分布外动作（out-of-distribution actions）导致的分布偏移。基于离线多智能体星际争霸II微操作任务的全面评估，我们证明了OMAC相较于现有最先进的离线多智能体RL方法具有更优越的性能。