End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practice. There have been some early attempts to use attention maps or cost volume for better model explainability which is difficult for ordinary passengers to understand. To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. ADAPT jointly trains both the driving caption task and the vehicular control prediction task, through a shared video representation. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time. The code, models and data are available at https://github.com/jxbbb/ADAPT.

翻译：端到端自动驾驶在交通行业具有巨大潜力。然而，自动决策过程缺乏透明度和可解释性阻碍了其在实际中的工业应用。已有一些早期工作尝试使用注意力图或代价体来提高模型可解释性，但普通乘客难以理解。为弥合这一差距，我们提出一种基于Transformer的端到端架构——ADAPT（动作感知的驾驶描述Transformer），该架构为自动驾驶车辆控制与动作的每个决策步骤提供用户友好的自然语言描述与推理。ADAPT通过共享的视频表示，联合训练驾驶描述任务与车辆控制预测任务。在BDD-X（伯克利深度驾驶解释）数据集上的实验表明，ADAPT框架在自动评估指标和人工评估中均达到了最先进水平。为证明所提框架在实际应用中的可行性，我们构建了一个新型可部署系统，该系统以原始车载视频为输入，实时输出动作描述与推理。代码、模型和数据可在https://github.com/jxbbb/ADAPT获取。