Overtaking on two-lane roads is a great challenge for autonomous vehicles, as oncoming traffic appearing on the opposite lane may require the vehicle to change its decision and abort the overtaking. Deep reinforcement learning (DRL) has shown promise for difficult decision problems such as this, but it requires massive number of data, especially if the action space is continuous. This paper proposes to incorporate guidance from an expert system into DRL to increase its sample efficiency in the autonomous overtaking setting. The guidance system developed in this study is composed of constrained iterative LQR and PID controllers. The novelty lies in the incorporation of a fading guidance function, which gradually decreases the effect of the expert system, allowing the agent to initially learn an appropriate action swiftly and then improve beyond the performance of the expert system. This approach thus combines the strengths of traditional control engineering with the flexibility of learning systems, expanding the capabilities of the autonomous system. The proposed methodology for autonomous vehicle overtaking does not depend on a particular DRL algorithm and three state-of-the-art algorithms are used as baselines for evaluation. Simulation results show that incorporating expert system guidance improves state-of-the-art DRL algorithms greatly in both sample efficiency and driving safety.

翻译：在两车道道路上进行超车对自动驾驶车辆而言是一项重大挑战，因为对向车道出现的来车可能迫使车辆改变决策并中止超车。深度强化学习在应对此类复杂决策问题时展现出潜力，但需要大量数据，尤其在连续动作空间场景下更为突出。本文提出将专家系统的指导融入深度强化学习，以提高其在自动驾驶超车场景中的样本效率。本研究开发的指导系统由带约束的迭代线性二次型调节器和PID控制器构成。其创新点在于引入渐弱指导函数，该函数逐步降低专家系统的影响，使智能体能够快速学习恰当动作，继而超越专家系统的性能表现。该方法融合了传统控制工程的优势与学习系统的灵活性，拓展了自动驾驶系统的能力边界。所提出的自动驾驶超车方法不依赖于特定深度强化学习算法，并采用三种前沿算法作为评估基准。仿真结果表明，融入专家系统指导能显著提升前沿深度强化学习算法的样本效率和驾驶安全性。