Tackling multi-agent learning problems efficiently is a challenging task in continuous action domains. While value-based algorithms excel in sample efficiency when applied to discrete action domains, they are usually inefficient when dealing with continuous actions. Policy-based algorithms, on the other hand, attempt to address this challenge by leveraging critic networks for guiding the learning process and stabilizing the gradient estimation. The limitations in the estimation of true return and falling into local optima in these methods result in inefficient and often sub-optimal policies. In this paper, we diverge from the trend of further enhancing critic networks, and focus on improving the effectiveness of value-based methods in multi-agent continuous domains by concurrently evaluating numerous actions. We propose a novel multi-agent value-based algorithm, Mixed Q-Functionals (MQF), inspired from the idea of Q-Functionals, that enables agents to transform their states into basis functions. Our algorithm fosters collaboration among agents by mixing their action-values. We evaluate the efficacy of our algorithm in six cooperative multi-agent scenarios. Our empirical findings reveal that MQF outperforms four variants of Deep Deterministic Policy Gradient through rapid action evaluation and increased sample efficiency.

翻译：在连续动作域中高效处理多智能体学习问题是一项具有挑战性的任务。尽管基于价值的算法在应用于离散动作域时展现出样本效率优势，但在处理连续动作时通常效率低下。而基于策略的算法通过利用评论家网络指导学习过程并稳定梯度估计来试图解决该挑战，但这些方法在真实回报估计以及陷入局部最优方面存在局限性，导致策略效率低下且往往次优。本文跳出进一步强化评论家网络的趋势，专注于通过并行评估大量动作来提升基于价值方法在多智能体连续域中的有效性。受Q泛函思想的启发，我们提出了一种新颖的多智能体基于价值算法——混合Q泛函（MQF），该算法使智能体能够将其状态转换为基函数。通过混合各智能体的动作价值，我们的算法促进了智能体间的协作。我们在六个协作式多智能体场景中评估了该算法的有效性。实证结果表明，MQF通过快速动作评估和更高的样本效率，优于深度确定性策略梯度（DDPG）的四种变体。