Deep cooperative multi-agent reinforcement learning has demonstrated its remarkable success over a wide spectrum of complex control tasks. However, recent advances in multi-agent learning mainly focus on value decomposition while leaving entity interactions still intertwined, which easily leads to over-fitting on noisy interactions between entities. In this work, we introduce a novel interactiOn Pattern disenTangling (OPT) method, to disentangle the entity interactions into interaction prototypes, each of which represents an underlying interaction pattern within a subgroup of the entities. OPT facilitates filtering the noisy interactions between irrelevant entities and thus significantly improves generalizability as well as interpretability. Specifically, OPT introduces a sparse disagreement mechanism to encourage sparsity and diversity among discovered interaction prototypes. Then the model selectively restructures these prototypes into a compact interaction pattern by an aggregator with learnable weights. To alleviate the training instability issue caused by partial observability, we propose to maximize the mutual information between the aggregation weights and the history behaviors of each agent. Experiments on single-task, multi-task and zero-shot benchmarks demonstrate that the proposed method yields results superior to the state-of-the-art counterparts. Our code is available at https://github.com/liushunyu/OPT.

翻译：深度协作式多智能体强化学习已在广泛复杂的控制任务中展现出卓越成效。然而，当前多智能体学习的研究进展主要集中于价值分解，而实体间的交互关系仍处于纠缠状态，这容易导致模型在实体间的噪声交互上产生过拟合。本研究提出了一种新颖的交互模式解耦方法，通过将实体交互分解为多个交互原型——每个原型代表实体子群内的一种潜在交互模式。该方法能有效过滤无关实体间的噪声交互，从而显著提升模型的泛化能力与可解释性。具体而言，OPT引入了一种稀疏分歧机制，以促进所发现交互原型间的稀疏性与多样性。随后，模型通过具有可学习权重的聚合器，选择性地将这些原型重组为紧凑的交互模式。为缓解部分可观测性导致的训练不稳定问题，我们提出最大化聚合权重与各智能体历史行为之间的互信息。在单任务、多任务及零样本基准测试上的实验表明，该方法取得了优于现有先进模型的性能。代码已开源：https://github.com/liushunyu/OPT。