In this work we consider a generalization of the well-known multivehicle routing problem: given a network, a set of agents occupying a subset of its nodes, and a set of tasks, we seek a minimum cost sequence of movements subject to the constraint that each task is visited by some agent at least once. The classical version of this problem assumes a central computational server that observes the entire state of the system perfectly and directs individual agents according to a centralized control scheme. In contrast, we assume that there is no centralized server and that each agent is an individual processor with no a priori knowledge of the underlying network (including task and agent locations). Moreover, our agents possess strictly local communication and sensing capabilities (restricted to a fixed radius around their respective locations), aligning more closely with several real-world multiagent applications. These restrictions introduce many challenges that are overcome through local information sharing and direct coordination between agents. We present a fully distributed, online, and scalable reinforcement learning algorithm for this problem whereby agents self-organize into local clusters and independently apply a multiagent rollout scheme locally to each cluster. We demonstrate empirically via extensive simulations that there exists a critical sensing radius beyond which the distributed rollout algorithm begins to improve over a greedy base policy. This critical sensing radius grows proportionally to the $\log^*$ function of the size of the network, and is, therefore, a small constant for any relevant network. Our decentralized reinforcement learning algorithm achieves approximately a factor of two cost improvement over the base policy for a range of radii bounded from below and above by two and three times the critical sensing radius, respectively.

翻译：本文考虑经典多车辆路径问题的泛化形式：给定网络、占据部分节点的智能体集合及任务集合，在满足每个任务至少被某个智能体访问一次的约束下，寻求最小代价的移动序列。经典版本假设存在中央计算服务器，能够完美观测系统全局状态并按照集中控制方案指导各智能体。与此相反，本文假设不存在中央服务器，且每个智能体均为独立处理器，对底层网络（包括任务与智能体位置）无先验知识。此外，智能体仅具备严格的局部通信与感知能力（限制在以各自位置为中心固定半径范围内），这更贴近现实中的多智能体应用场景。这些限制通过局部信息共享与智能体间直接协调得以克服。本文针对该问题提出了一种完全分布式、在线且可扩展的强化学习算法，使智能体能够自组织形成局部集群，并独立对各集群应用多智能体滚动优化方案。通过大量仿真实验，我们实证发现存在一个关键感知半径，超过该半径后分布式滚动优化算法开始优于贪心基线策略。该关键感知半径与网络规模的log*函数成正比增长，因此对任意实际网络而言均为小常数。在关键感知半径二倍至三倍的下界与上界范围内，我们的分布式强化学习算法相较于基线策略实现了约两倍的成本改善。