World models have recently attracted growing interest in Multi-Agent Reinforcement Learning (MARL) due to their ability to improve sample efficiency for policy learning. However, accurately modeling environments in MARL is challenging due to the exponentially large joint action space and highly uncertain dynamics inherent in multi-agent systems. To address this, we reduce modeling complexity by shifting from jointly modeling the entire state-action transition dynamics to focusing on the state space alone at each timestep through sequential agent modeling. Specifically, our approach enables the model to progressively resolve uncertainty while capturing the structured dependencies among agents, providing a more accurate representation of how agents influence the state. Interestingly, this sequential revelation of agents' actions in a multi-agent system aligns with the reverse process in diffusion models--a class of powerful generative models known for their expressiveness and training stability compared to autoregressive or latent variable models. Leveraging this insight, we develop a flexible and robust world model for MARL using diffusion models. Our method, Diffusion-Inspired Multi-Agent world model (DIMA), achieves state-of-the-art performance across multiple multi-agent control benchmarks, significantly outperforming prior world models in terms of final return and sample efficiency, including MAMuJoCo and Bi-DexHands. DIMA establishes a new paradigm for constructing multi-agent world models, advancing the frontier of MARL research. Codes are open-sourced at https://github.com/breez3young/DIMA.

翻译：世界模型因其能够提升策略学习的样本效率，在多智能体强化学习领域日益受到关注。然而，由于多智能体系统固有的指数级联合动作空间和高度不确定的动态特性，在MARL中精确建模环境具有挑战性。为解决这一问题，我们通过序列化智能体建模，将建模重点从联合建模整个状态-动作转移动态，转向每个时间步仅关注状态空间，从而降低建模复杂度。具体而言，我们的方法使模型能够在捕捉智能体间结构化依赖关系的同时，逐步消解不确定性，从而更准确地表征智能体如何影响状态。有趣的是，这种多智能体系统中智能体动作的序列化揭示过程，与扩散模型的反向过程具有内在一致性——扩散模型作为一类强大的生成模型，相较于自回归或隐变量模型，以其表达能力和训练稳定性著称。基于这一洞见，我们利用扩散模型为MARL开发了一个灵活且鲁棒的世界模型。我们的方法——扩散启发的多智能体世界模型，在多个多智能体控制基准测试中取得了最先进的性能，在最终回报和样本效率方面显著优于先前世界模型，包括MAMuJoCo和Bi-DexHands。DIMA为构建多智能体世界模型建立了新范式，推动了MARL研究的前沿。代码已在https://github.com/breez3young/DIMA开源。