Cooperative multi-agent reinforcement learning (MARL) is a challenging task, as agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle to distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL. By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also adapting to achieve more muscular efficiency and fairness when diversity is observed.Our extensive experiments on popular benchmarks validate the effectiveness of our theory and method, demonstrating significant improvements in learning efficiency and fairness.

翻译：合作式多智能体强化学习是一项具有挑战性的任务，因为智能体必须从共享的团队奖励中学习复杂且多样化的个体策略。然而，现有方法难以区分并利用重要的个体经验，因为它们缺乏将团队奖励有效分解为个体奖励的途径。为应对这一挑战，我们提出了DIFFER——一个用于分解个体奖励以支持多智能体强化学习中公平经验回放的强大理论框架。通过强制网络梯度不变性，我们建立了一个偏微分方程，其解能够推导出潜在的个体奖励函数。进而可从求解得到的闭式个体奖励中计算个体TD误差，该误差指示了每条经验在学习任务中的重要性，并指导训练过程。当个体经验同质时，我们的方法优雅地实现了与原始学习框架的等价性；同时，在观测到多样性时，也能自适应地实现更强的效率与公平性。我们在广泛使用的基准测试上进行了大量实验，验证了所提理论与方法的有效性，结果显示其在学习效率与公平性方面均有显著提升。