Reinforcement learning has been used to train policies that outperform even the best human players in various games. However, a large amount of data is needed to achieve good performance, which in turn requires building large-scale frameworks and simulators. In this paper, we study how large-scale reinforcement learning can be applied to autonomous driving, analyze how the resulting policies perform as the experiment size is scaled, and what the most important factors contributing to policy performance are. To do this, we first introduce a hardware-accelerated autonomous driving simulator, which allows us to efficiently collect experience from billions of agent steps. This simulator is paired with a large-scale, multi-GPU reinforcement learning framework. We demonstrate that simultaneous scaling of dataset size, model size, and agent steps trained provides increasingly strong driving policies in regard to collision, traffic rule violations, and progress. In particular, our best policy reduces the failure rate by 57% while improving progress by 23% compared to the current state-of-the-art machine learning policies for autonomous driving.

翻译：强化学习已被用于训练出在多种游戏中超越最优人类玩家的策略。然而，实现优异性能需要海量数据，这进而要求构建大规模框架与仿真器。本文研究如何将大规模强化学习应用于自动驾驶，分析策略性能随实验规模扩展的变化规律，并探究影响策略性能的最关键因素。为此，我们首先引入一款硬件加速的自动驾驶仿真器，可高效采集数十亿智能体步数的经验数据。该仿真器与大规模多GPU强化学习框架协同运作。研究表明，同步扩展数据集规模、模型大小及智能体训练步数，可在碰撞率、交通规则违反率和通行效率方面持续提升策略性能。特别地，与当前最先进的自动驾驶机器学习策略相比，我们的最优策略将失误率降低57%，同时使通行效率提升23%。