More research attention has recently been given to end-to-end autonomous driving technologies where the entire driving pipeline is replaced with a single neural network because of its simpler structure and faster inference time. Despite this appealing approach largely reducing the components in the driving pipeline, its simplicity also leads to interpretability problems and safety issues. The trained policy is not always compliant with the traffic rules and it is also hard to discover the reason for the misbehavior because of the lack of intermediate outputs. Meanwhile, sensors are also critical to autonomous driving's security and feasibility to perceive the surrounding environment under complex driving scenarios. In this paper, we proposed P-CSG, a penalty-based imitation learning approach with cross semantics generation sensor fusion technologies to increase the overall performance of end-to-end autonomous driving. In this method, we introduce three penalties - red light, stop sign, and curvature speed penalty to make the agent more sensitive to traffic rules. The proposed cross semantics generation helps to align the shared information from different input modalities. We assessed our model's performance using the CARLA leaderboard - Town 05 Long benchmark and Longest6 Benchmark, achieving an impressive driving score improvement. Furthermore, we conducted robustness evaluations against adversarial attacks like FGSM and Dot attacks, revealing a substantial increase in robustness compared to baseline models. More detailed information, such as code base resources, and videos can be found at https://hk-zh.github.io/p-csg-plus.

翻译：近年来，端到端自动驾驶技术因其结构简单、推理速度快而受到更多研究关注，该技术用单个神经网络替代了整个驾驶流程。尽管这种引人注目的方法大幅减少了驾驶流程中的组件，但其简洁性也带来了可解释性问题与安全隐患。训练得到的策略并非始终遵守交通规则，且由于缺乏中间输出，难以发现行为异常的原因。同时，传感器对于自动驾驶在复杂驾驶场景中感知周围环境的安全性和可行性也至关重要。本文提出P-CSG——一种基于惩罚的模仿学习方法，结合跨语义生成传感器融合技术，以提升端到端自动驾驶的整体性能。该方法引入三种惩罚机制——红灯惩罚、停车标志惩罚和弯道速度惩罚，使智能体对交通规则更加敏感。所提出的跨语义生成有助于对齐来自不同输入模态的共享信息。我们使用CARLA排行榜的Town 05 Long基准测试和Longest6基准测试评估模型性能，取得了显著的驾驶分数提升。此外，我们针对FGSM和Dot攻击等对抗性攻击进行了鲁棒性评估，结果显示与基线模型相比鲁棒性大幅提升。更多详细信息（如代码库资源和视频）可访问https://hk-zh.github.io/p-csg-plus。