Centralized Training with Decentralized Execution (CTDE) has been proven to be an effective paradigm in cooperative multi-agent reinforcement learning (MARL). One of the major challenges is credit assignment, which aims to credit agents by their contributions. While prior studies have shown great success, their methods typically fail to work in episodic reinforcement learning scenarios where global rewards are revealed only at the end of the episode. They lack the functionality to model complicated relations of the delayed global reward in the temporal dimension and suffer from inefficiencies. To tackle this, we introduce Spatial-Temporal Attention with Shapley (STAS), a novel method that learns credit assignment in both temporal and spatial dimensions. It first decomposes the global return back to each time step, then utilizes the Shapley Value to redistribute the individual payoff from the decomposed global reward. To mitigate the computational complexity of the Shapley Value, we introduce an approximation of marginal contribution and utilize Monte Carlo sampling to estimate it. We evaluate our method on an Alice & Bob example and MPE environments across different scenarios. Our results demonstrate that our method effectively assigns spatial-temporal credit, outperforming all state-of-the-art baselines.

翻译：摘要：集中训练与分散执行（CTDE）已被证明是协作式多智能体强化学习（MARL）中的有效范式。其中主要挑战之一为信用分配，旨在根据智能体的贡献对其进行归责。尽管先前研究取得了显著成功，但其方法通常无法适用于仅在回合结束时揭示全局奖励的回合制强化学习场景。这些方法缺乏对延迟全局奖励在时间维度上复杂关系的建模能力，且存在效率低下的问题。为解决此问题，我们提出了基于夏普利值的时空注意力机制（STAS），这是一种在时间与空间维度上同时学习信用分配的新方法。该方法首先将全局回报分解至每个时间步，随后利用夏普利值从分解后的全局奖励中重新分配个体收益。为降低夏普利值的计算复杂度，我们引入了边际贡献的近似方法，并利用蒙特卡洛采样进行估计。我们在Alice与Bob算例及多粒子环境（MPE）的多场景下评估了该方法。结果表明，我们的方法能够有效分配时空信用，并在性能上超越所有最先进的基线模型。