Deep reinforcement learning methods have shown great performance on many challenging cooperative multi-agent tasks. Two main promising research directions are multi-agent value function decomposition and multi-agent policy gradients. In this paper, we propose a new decomposed multi-agent soft actor-critic (mSAC) method, which incorporates the idea of the multi-agent value function decomposition and soft policy iteration framework effectively and is a combination of novel and existing techniques, including decomposed Q value network architecture, decentralized probabilistic policy, and counterfactual advantage function (optional). Theoretically, mSAC supports efficient off-policy learning and addresses credit assignment problem partially in both discrete and continuous action spaces. Tested on StarCraft II micromanagement cooperative multiagent benchmark, we empirically investigate the performance of mSAC against its variants and analyze the effects of the different components. Experimental results demonstrate that mSAC significantly outperforms policy-based approach COMA, and achieves competitive results with SOTA value-based approach Qmix on most tasks in terms of asymptotic perfomance metric. In addition, mSAC achieves pretty good results on large action space tasks, such as 2c_vs_64zg and MMM2.

翻译：深度强化学习方法在许多具有挑战性的多代理人合作性任务中表现出了巨大的业绩。两个主要有希望的研究方向是多试剂价值功能分解和多剂政策梯度。在本文件中,我们提出了一个新的分解多剂软性行为者-加速(mSAC)方法,其中包括多剂价值功能分解和软政策复制框架的概念,并有效地结合了新颖和现有技术,包括分解的Q值网络结构、分散的概率政策和反现实优势功能(可选择的)。理论上,MSAC支持高效的离散政策学习和部分在离散和连续行动空间解决信用分配问题。在StarCraft II微型管理合作性多剂基准上测试了多剂作用分解多剂功能分解和软性政策循环框架的构想,并有效地分析了不同组成部分的效应。实验结果表明,MSAAC大大偏离了以政策为基础的COMA方法,并且通过SATA增值方法取得了竞争性结果。在最大任务上,例如大型的perfomace-MMMM2等成果。