Despite significant progress in autonomous vehicles (AVs), the development of driving policies that ensure both the safety of AVs and traffic flow efficiency has not yet been fully explored. In this paper, we propose an enhanced human-in-the-loop reinforcement learning method, termed the Human as AI mentor-based deep reinforcement learning (HAIM-DRL) framework, which facilitates safe and efficient autonomous driving in mixed traffic platoon. Drawing inspiration from the human learning process, we first introduce an innovative learning paradigm that effectively injects human intelligence into AI, termed Human as AI mentor (HAIM). In this paradigm, the human expert serves as a mentor to the AI agent. While allowing the agent to sufficiently explore uncertain environments, the human expert can take control in dangerous situations and demonstrate correct actions to avoid potential accidents. On the other hand, the agent could be guided to minimize traffic flow disturbance, thereby optimizing traffic flow efficiency. In detail, HAIM-DRL leverages data collected from free exploration and partial human demonstrations as its two training sources. Remarkably, we circumvent the intricate process of manually designing reward functions; instead, we directly derive proxy state-action values from partial human demonstrations to guide the agents' policy learning. Additionally, we employ a minimal intervention technique to reduce the human mentor's cognitive load. Comparative results show that HAIM-DRL outperforms traditional methods in driving safety, sampling efficiency, mitigation of traffic flow disturbance, and generalizability to unseen traffic scenarios. The code and demo videos for this paper can be accessed at: https://zilin-huang.github.io/HAIM-DRL-website/}{https://zilin-huang.github.io/HAIM-DRL-website/.

翻译：尽管自动驾驶汽车取得了显著进展，但尚未充分探索如何制定既能保障自动驾驶汽车安全又能提升交通流效率的驾驶策略。本文提出一种增强型人在环强化学习方法，即基于人类AI导师的深度强化学习框架（HAIM-DRL），以促进混合交通编队中的安全高效自动驾驶。受人类学习过程启发，我们首先引入一种创新学习范式——人类作为AI导师（HAIM），该范式有效将人类智能注入人工智能。在此范式中，人类专家担任AI智能体的导师。在允许智能体充分探索不确定环境的同时，人类专家可在危险情况下接管控制权，通过示范正确动作避免潜在事故；另一方面，智能体可被引导以最小化交通流扰动，从而优化交通流效率。具体而言，HAIM-DRL将自由探索收集的数据与部分人类示范数据作为两类训练来源。值得注意的是，我们规避了人工设计奖励函数的复杂流程，而是直接从部分人类示范中推导代理状态-动作值来指导智能体的策略学习。此外，我们采用最小干预技术以减轻人类导师的认知负荷。对比结果表明，HAIM-DRL在驾驶安全性、采样效率、交通流扰动抑制能力及对未见交通场景的泛化性方面均优于传统方法。本文代码与演示视频可访问：https://zilin-huang.github.io/HAIM-DRL-website/。