The paper considers independent reinforcement learning (IRL) for multi-agent collaborative decision-making in the paradigm of federated learning (FL). However, FL generates excessive communication overheads between agents and a remote central server, especially when it involves a large number of agents or iterations. Besides, due to the heterogeneity of independent learning environments, multiple agents may undergo asynchronous Markov decision processes (MDPs), which will affect the training samples and the model's convergence performance. On top of the variation-aware periodic averaging (VPA) method and the policy-based deep reinforcement learning (DRL) algorithm (i.e., proximal policy optimization (PPO)), this paper proposes two advanced optimization schemes orienting to stochastic gradient descent (SGD): 1) A decay-based scheme gradually decays the weights of a model's local gradients with the progress of successive local updates, and 2) By representing the agents as a graph, a consensus-based scheme studies the impact of exchanging a model's local gradients among nearby agents from an algebraic connectivity perspective. This paper also provides novel convergence guarantees for both developed schemes, and demonstrates their superior effectiveness and efficiency in improving the system's utility value through theoretical analyses and simulation results.

翻译：本文考虑联邦学习（FL）范式下多智能体协作决策中的独立强化学习（IRL）。然而，FL在智能体与远程中央服务器之间产生过大的通信开销，尤其是在涉及大量智能体或迭代次数时。此外，由于独立学习环境的异质性，多智能体可能经历异步马尔可夫决策过程（MDP），这将影响训练样本及模型的收敛性能。基于方差感知周期平均（VPA）方法和基于策略的深度强化学习（DRL）算法（即近端策略优化（PPO）），本文针对随机梯度下降（SGD）提出了两种先进的优化方案：1）一种基于衰减的方案，随着连续局部更新的推进逐渐衰减模型局部梯度的权重；2）通过将智能体表示为图，一种基于共识的方案从代数连通性角度研究相邻智能体之间交换模型局部梯度的影响。本文还为这两种开发方案提供了新颖的收敛保证，并通过理论分析和仿真结果证明了它们在提升系统效用价值方面的卓越有效性和效率。