A central question in robotics is how to design a control system for an agile mobile robot. This paper studies this question systematically, focusing on a challenging setting: autonomous drone racing. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. We then investigated which fundamental factors have contributed to the success of RL or have limited OC. Our study indicates that the fundamental advantage of RL over OC is not that it optimizes its objective better but that it optimizes a better objective. OC decomposes the problem into planning and control with an explicit intermediate representation, such as a trajectory, that serves as an interface. This decomposition limits the range of behaviors that can be expressed by the controller, leading to inferior control performance when facing unmodeled effects. In contrast, RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses. Our findings allowed us to push an agile drone to its maximum performance, achieving a peak acceleration greater than 12 times the gravitational acceleration and a peak velocity of 108 kilometers per hour. Our policy achieved superhuman control within minutes of training on a standard workstation. This work presents a milestone in agile robotics and sheds light on the role of RL and OC in robot control.

翻译：机器人学中的一个核心问题是如何为敏捷移动机器人设计控制系统。本文系统研究了这一问题，聚焦于具有挑战性的场景：自主无人机竞速。我们展示了在该场景下，经强化学习训练的神经网络控制器优于最优控制方法。随后探究了促成RL成功或限制OC发挥的根本因素。研究表明，RL相较于OC的根本优势并非在于其能更好地优化目标，而在于它优化了更优的目标。OC通过显式中间表征（如轨迹）将问题分解为规划与控制两个阶段，这种界面的存在限制了控制器可表达的行为范围，导致面对未建模效应时控制性能不佳。相比之下，RL能直接优化任务级目标，并利用域随机化应对模型不确定性，从而发现更鲁棒的控制响应。研究结果使我们得以将敏捷无人机性能推向极限，实现了超过12倍重力加速度的峰值加速度与108公里/小时的极限速度。在标准工作站上仅需数分钟训练，我们的策略即能达到超人类控制水平。本工作标志着敏捷机器人领域的里程碑，阐明了RL与OC在机器人控制中的角色定位。