Despite significant progress in autonomous vehicles (AVs), the development of driving policies that ensure both the safety of AVs and traffic flow efficiency has not yet been fully explored. In this paper, we propose an enhanced human-in-the-loop reinforcement learning method, termed the Human as AI mentor-based deep reinforcement learning (HAIM-DRL) framework, which facilitates safe and efficient autonomous driving in mixed traffic platoon. Drawing inspiration from the human learning process, we first introduce an innovative learning paradigm that effectively injects human intelligence into AI, termed Human as AI mentor (HAIM). In this paradigm, the human expert serves as a mentor to the AI agent. While allowing the agent to sufficiently explore uncertain environments, the human expert can take control in dangerous situations and demonstrate correct actions to avoid potential accidents. On the other hand, the agent could be guided to minimize traffic flow disturbance, thereby optimizing traffic flow efficiency. In detail, HAIM-DRL leverages data collected from free exploration and partial human demonstrations as its two training sources. Remarkably, we circumvent the intricate process of manually designing reward functions; instead, we directly derive proxy state-action values from partial human demonstrations to guide the agents' policy learning. Additionally, we employ a minimal intervention technique to reduce the human mentor's cognitive load. Comparative results show that HAIM-DRL outperforms traditional methods in driving safety, sampling efficiency, mitigation of traffic flow disturbance, and generalizability to unseen traffic scenarios. The code and demo videos for this paper can be accessed at: https://zilin-huang.github.io/HAIM-DRL-website/

翻译：尽管自动驾驶汽车（AV）研究取得显著进展，但尚未充分探索能同时保障车辆安全与交通流效率的驾驶策略。本文提出一种增强型人机协同强化学习方法——基于人类AI导师的深度强化学习框架（HAIM-DRL），旨在实现混合交通队列中的安全高效自动驾驶。受人类学习过程启发，我们首先引入创新学习范式“人类作为AI导师”（HAIM），有效将人类智能注入人工智能系统。在该范式中，人类专家作为AI智能体的导师，在允许智能体充分探索不确定环境的同时，可在危险场景下接管控制并演示正确动作以规避潜在事故；另一方面，智能体可接受引导以最小化交通流扰动，从而优化交通流效率。具体而言，HAIM-DRL将自由探索数据与部分人类示教数据作为双重训练来源。值得关注的是，我们绕过了手动设计奖励函数的复杂过程，转而直接从部分人类示教数据中推导代理状态-动作值来引导智能体策略学习。此外，我们采用最小干预技术减轻人类导师的认知负荷。对比结果表明，HAIM-DRL在驾驶安全性、采样效率、交通流扰动抑制能力及对未见交通场景的泛化性方面均优于传统方法。本文代码与演示视频可通过以下链接获取：https://zilin-huang.github.io/HAIM-DRL-website/