Recent accidents involving self-driving cars call for extensive testing efforts to improve the safety and robustness of autonomous driving. However, constructing test scenarios for autonomous driving is tedious and time-consuming. In this work, we develop an end-to-end test generation framework called TARGET, which automatically constructs test scenarios from human-written traffic rules in an autonomous driving simulator. To handle the ambiguity and sophistication of natural language, TARGET uses GPT-3 to extract key information related to the test scenario from a traffic rule and represents the extracted information in a test scenario schema. Then, TARGET synthesizes the corresponding scenario scripts to construct the test scenario based on the scenario representation. We have evaluated TARGET on four autonomous driving systems, 18 traffic rules, and 8 road maps. TARGET can successfully generate 75 test scenarios and detect 247 traffic rule violations. Based on the violation logs (e.g., waypoints of ego vehicles), we are able to identify three underlying issues in these autonomous driving systems, which are either confirmed by the developers or the existing bug reports.
翻译:近期涉及自动驾驶汽车的事故要求进行大量测试工作以提高自动驾驶的安全性和鲁棒性。然而,为自动驾驶构建测试场景既繁琐又耗时。在这项工作中,我们开发了一个名为TARGET的端到端测试生成框架,该框架能自动从人工编写的交通规则中在自动驾驶模拟器中构建测试场景。为处理自然语言的歧义性和复杂性,TARGET使用GPT-3从交通规则中提取与测试场景相关的关键信息,并将提取的信息表示为测试场景模式。随后,TARGET基于场景表示合成相应的场景脚本来构建测试场景。我们在四个自动驾驶系统、18条交通规则和8张道路地图上对TARGET进行了评估。TARGET成功生成了75个测试场景并检测到247项交通规则违反行为。基于违反日志(例如自车的路径点),我们能够识别这些自动驾驶系统中的三个潜在问题,这些问题已得到开发者确认或与现有错误报告相符。