MADDPG is an algorithm in multi-agent reinforcement learning (MARL) that extends the popular single-agent method, DDPG, to multi-agent scenarios. Importantly, DDPG is an algorithm designed for continuous action spaces, where the gradient of the state-action value function exists. For this algorithm to work in discrete action spaces, discrete gradient estimation must be performed. For MADDPG, the Gumbel-Softmax (GS) estimator is used -- a reparameterisation which relaxes a discrete distribution into a similar continuous one. This method, however, is statistically biased, and a recent MARL benchmarking paper suggests that this bias makes MADDPG perform poorly in grid-world situations, where the action space is discrete. Fortunately, many alternatives to the GS exist, boasting a wide range of properties. This paper explores several of these alternatives and integrates them into MADDPG for discrete grid-world scenarios. The corresponding impact on various performance metrics is then measured and analysed. It is found that one of the proposed estimators performs significantly better than the original GS in several tasks, achieving up to 55% higher returns, along with faster convergence.

翻译：MADDPG是一种多智能体强化学习（MARL）算法，它将流行的单智能体方法DDPG扩展到多智能体场景。关键在于，DDPG是针对连续动作空间设计的算法，其中状态-动作值函数的梯度存在。为使该算法在离散动作空间中有效运行，必须进行离散梯度估计。对于MADDPG，采用了Gumbel-Softmax（GS）估计器——一种将离散分布松弛为类似连续分布的重参数化方法。然而，该方法存在统计偏差，且近期一篇MARL基准测试论文指出，这种偏差导致MADDPG在动作空间为离散的网格世界场景中表现不佳。幸运的是，存在许多GS的替代方法，它们具有广泛的特性。本文探索了其中几种替代方法，并将其整合到面向离散网格世界场景的MADDPG中。随后，测量并分析了这些方法对各种性能指标的影响。结果发现，在多个任务中，所提出的一种估计器的表现显著优于原始GS，实现了高达55%的回报率提升，并具有更快的收敛速度。