This work considers multiple agents traversing a network from a source node to the goal node. The cost to an agent for traveling a link has a private as well as a congestion component. The agent's objective is to find a path to the goal node with minimum overall cost in a decentralized way. We model this as a fully decentralized multi-agent reinforcement learning problem and propose a novel multi-agent congestion cost minimization (MACCM) algorithm. Our MACCM algorithm uses linear function approximations of transition probabilities and the global cost function. In the absence of a central controller and to preserve privacy, agents communicate the cost function parameters to their neighbors via a time-varying communication network. Moreover, each agent maintains its estimate of the global state-action value, which is updated via a multi-agent extended value iteration (MAEVI) sub-routine. We show that our MACCM algorithm achieves a sub-linear regret. The proof requires the convergence of cost function parameters, the MAEVI algorithm, and analysis of the regret bounds induced by the MAEVI triggering condition for each agent. We implement our algorithm on a two node network with multiple links to validate it. We first identify the optimal policy, the optimal number of agents going to the goal node in each period. We observe that the average regret is close to zero for 2 and 3 agents. The optimal policy captures the trade-off between the minimum cost of staying at a node and the congestion cost of going to the goal node. Our work is a generalization of learning the stochastic shortest path problem.

翻译：本文考虑多个智能体从源节点到目标节点穿越网络的问题。智能体在链路上行驶的成本包含私有成本与拥塞成本两部分。智能体的目标是通过去中心化方式找到通往目标节点的最小总成本路径。我们将此建模为完全去中心化的多智能体强化学习问题，并提出一种新型多智能体拥塞成本最小化（MACCM）算法。该算法采用转移概率和全局成本函数的线性函数逼近。在无中央控制器且需保护隐私的前提下，智能体通过时变通信网络向邻居传递成本函数参数。此外，每个智能体维护对全局状态-动作价值的估计，并通过多智能体扩展值迭代（MAEVI）子程序进行更新。我们证明MACCM算法可实现次线性遗憾值。证明过程需要确保成本函数参数收敛、MAEVI算法收敛，并分析由各智能体MAEVI触发条件所导致的遗憾界。我们在含多条链路的双节点网络上实现该算法以进行验证。首先确定最优策略及每个周期内前往目标节点的最优智能体数量。实验表明，当智能体数量为2和3时，平均遗憾值趋近于零。最优策略体现了节点停留最小成本与前往目标节点拥塞成本之间的权衡关系。本工作是对学习随机最短路径问题的一般性推广。