Deep Reinforcement Learning (DRL) has exhibited efficacy in resolving the Local Path Planning (LPP) problem. However, such application in the real world is immensely limited due to the deficient training efficiency and generalization capability of DRL. To alleviate these two issues, a solution named Color is proposed, which consists of an Actor-Sharer-Learner (ASL) training framework and a mobile robot-oriented simulator Sparrow. Specifically, the ASL intends to improve the training efficiency of DRL algorithms. It employs a Vectorized Data Collection (VDC) mode to expedite data acquisition, decouples the data collection from model optimization by multithreading, and partially connects the two procedures by harnessing a Time Feedback Mechanism (TFM) to evade data underuse or overuse. Meanwhile, the Sparrow simulator utilizes a 2D grid-based world, simplified kinematics, and conversion-free data flow to achieve a lightweight design. The lightness facilitates vectorized diversity, allowing diversified simulation setups across extensive copies of the vectorized environments, resulting in a notable enhancement in the generalization capability of the DRL algorithm being trained. Comprehensive experiments, comprising 57 DRL benchmark environments, 32 simulated and 36 real-world LPP scenarios, have been conducted to corroborate the superiority of our method in terms of efficiency and generalization. The code and the video of this paper are accessible at https://github.com/XinJingHao/Color.
翻译:深度强化学习(DRL)在解决局部路径规划(LPP)问题中展现出有效性。然而,由于DRL训练效率低下且泛化能力不足,其在真实世界的应用受到极大限制。为缓解这两个问题,本文提出名为Color的解决方案,包含参与者-共享者-学习者(ASL)训练框架和面向移动机器人的Sparrow仿真器。具体而言,ASL旨在提升DRL算法的训练效率:它采用向量化数据采集(VDC)模式加速数据获取,通过多线程将数据采集与模型优化解耦,并利用时间反馈机制(TFM)部分连接这两个过程以避免数据欠用或过用。同时,Sparrow仿真器采用基于2D网格的世界模型、简化运动学模型和无转换数据流实现轻量化设计。这种轻量化特性促进了向量化多样性,允许在大量向量化环境副本中部署多样化的仿真配置,从而显著增强所训练DRL算法的泛化能力。我们开展了包含57个DRL基准环境、32个仿真场景和36个真实世界LPP场景的综合实验,验证了本方法在效率和泛化性方面的优越性。本文代码与视频见https://github.com/XinJingHao/Color。