We propose an end-to-end learning framework based on hierarchical reinforcement learning, called H-TSP, for addressing the large-scale Travelling Salesman Problem (TSP). The proposed H-TSP constructs a solution of a TSP instance starting from the scratch relying on two components: the upper-level policy chooses a small subset of nodes (up to 200 in our experiment) from all nodes that are to be traversed, while the lower-level policy takes the chosen nodes as input and outputs a tour connecting them to the existing partial route (initially only containing the depot). After jointly training the upper-level and lower-level policies, our approach can directly generate solutions for the given TSP instances without relying on any time-consuming search procedures. To demonstrate effectiveness of the proposed approach, we have conducted extensive experiments on randomly generated TSP instances with different numbers of nodes. We show that H-TSP can achieve comparable results (gap 3.42% vs. 7.32%) as SOTA search-based approaches, and more importantly, we reduce the time consumption up to two orders of magnitude (3.32s vs. 395.85s). To the best of our knowledge, H-TSP is the first end-to-end deep reinforcement learning approach that can scale to TSP instances of up to 10000 nodes. Although there are still gaps to SOTA results with respect to solution quality, we believe that H-TSP will be useful for practical applications, particularly those that are time-sensitive e.g., on-call routing and ride hailing service.

翻译：我们提出了一种基于分层强化学习的端到端学习框架，称为H-TSP，用于解决大规模旅行商问题（TSP）。所提出的H-TSP从零开始构建TSP实例的解，其依赖两个组成部分：上层策略从所有待遍历节点中选择一个小子集（实验中最多200个节点），而下层策略以所选节点为输入，输出一条连接这些节点到现有部分路径（初始仅包含仓库）的巡回路线。通过联合训练上下层策略，我们的方法可直接为给定TSP实例生成解，无需依赖任何耗时的搜索过程。为证明所提出方法的有效性，我们在不同节点数目的随机生成TSP实例上进行了广泛实验。结果表明，H-TSP可达到与基于搜索的最先进方法相当的结果（差距3.42% vs. 7.32%），更重要的是，我们将时间消耗降低了两个数量级（3.32秒 vs. 395.85秒）。据我们所知，H-TSP是首个能够扩展至多达10000个节点TSP实例的端到端深度强化学习方法。尽管在解质量方面与最先进结果仍存在差距，但我们相信H-TSP将对实际应用具有价值，特别是对时间敏感的场景，如即时路由和网约车服务。