The increasing levels of software- and data-intensive driving automation call for an evolution of automotive software testing. As a recommended practice of the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision detection system) using two different commercial simulators, namely, TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we compare critical test scenarios generated using our SBST solution in these two simulators. We show that SBST can be used to effectively and efficiently generate critical test scenarios in both simulators, and the test results obtained from the two simulators can reveal several weaknesses of the ADAS under test. However, executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs, in particular, related to (1) safety violations revealed by tests, and (2) dynamics of cars and pedestrians. Based on our findings, we recommend future V&V plans to include multiple simulators to support robust simulation-based testing and to base test objectives on measures that are less dependant on the internals of the simulators.

翻译：越来越多的软件和数据密集型驱动自动化水平要求汽车软件测试的演化。作为ISO/PAS 21448(公路车辆预定功能的候选安全标准)ISO/PAS 21448的核查和校验(V&V)程序的建议做法,模拟测试有可能降低风险和成本。使用高级司机协助系统模拟器(ADS)设计测试自动化技术的研究越来越多。然而,如果在不同模拟器中执行同样的测试方案,结果又如何相似?我们进行一项复制性研究,将搜索软件测试(SBST)的解决方案应用到一个真实的ADAS(PeVi,行人视觉探测系统),采用两种不同的商业模拟器,即TASS/Siemens Prescan和ESI Pro-SiVIC。在最小的场景中,我们用我们的SBST解决方案在这两个模拟器中产生的关键测试情景。 我们显示,SBST可以有效和高效地在两个模拟器中生成关键测试方案。 在两个模拟器的模拟器的模拟器上,从两个测试结果中,从两个测试结果的测试结果到两个测试结果中,在两个测试结果中,在两个测试结果中,在两个测试结果中,在测试结果中,在两个测试结果中,在两个测试结果中,在两个测试结果中,在测试结果中,在测试结果中,在两个测试结果中,在两个测试结果中,在测试结果中,在测试结果中,在测试结果中,在测试结果中,在两个测试结果中,在测试结果中,在测试结果中,在测试结果中,在测试结果中,在两个是相同的是相同的。