This paper studies networked multi-agent reinforcement learning (NMARL) with interdependent rewards and coupled policies. In this setting, each agent's reward depends on its own state-action pair as well as those of its direct neighbors, and each agent's policy is parameterized by its local parameters together with those of its $κ_{p}$-hop neighbors, with $κ_{p}\geq 1$ denoting the coupled radius. To address the challenge of interdependent policies in collaborative optimization, we introduce a novel concept termed the ``neighbors' averaged $Q$-function'' and derive a new expression for the coupled-policy gradient. Based on these theoretical foundations, we develop a distributed scalable coupled policy (DSCP) algorithm, where each agent relies only on the state-action pairs of its $κ_{p}$-hop neighbors and the rewards of its $(κ_{p}+1)$-hop neighbors. Specifically, in the DSCP algorithm, we employ a geometric 2-horizon sampling method that does not require storing a full $Q$-table to obtain an unbiased estimate of the surrogate coupled-policy gradient under the executed policy. Moreover, each agent interacts exclusively with its direct neighbors to obtain accurate policy parameters, while maintaining local estimates of other agents' parameters to execute its local policy and collect samples for optimization. These estimates and policy parameters are updated in a distributed manner through local information exchange among direct neighbors, enabling coordinated policy updates across the network. We prove that the gradient norm of the objective function along the joint policy sequence produced by the proposed algorithm converges to zero almost surely. Finally, the effectiveness of the DSCP algorithm is demonstrated through simulations in a robot path planning environment, showing clear improvement over the compared algorithms in the tested setting.

翻译：暂无翻译