Rigorously testing autonomy systems is essential for making safe self-driving vehicles (SDV) a reality. It requires one to generate safety critical scenarios beyond what can be collected safely in the world, as many scenarios happen rarely on public roads. To accurately evaluate performance, we need to test the SDV on these scenarios in closed-loop, where the SDV and other actors interact with each other at each timestep. Previously recorded driving logs provide a rich resource to build these new scenarios from, but for closed loop evaluation, we need to modify the sensor data based on the new scene configuration and the SDV's decisions, as actors might be added or removed and the trajectories of existing actors and the SDV will differ from the original log. In this paper, we present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and converts it into a realistic closed-loop multi-sensor simulation. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene, and composites them together to simulate LiDAR and camera data at new viewpoints, with actors added or removed and at new placements. To better handle extrapolated views, we incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions. Our experiments show UniSim can simulate realistic sensor data with small domain gap on downstream tasks. With UniSim, we demonstrate closed-loop evaluation of an autonomy system on safety-critical scenarios as if it were in the real world.

翻译：严格测试自主系统对于实现安全自动驾驶车辆（SDV）至关重要。这需要生成无法通过安全采集真实世界数据获得的安全关键场景——因为许多场景在公共道路上极少发生。为准确评估性能，必须在闭环条件下测试SDV，即每个时间步中SDV与其他交通参与者相互交互。先前记录的驾驶日志为构建这类新场景提供了丰富资源，但闭环评估要求根据新场景配置和SDV决策修改传感器数据——因为可能新增或移除交通参与者，且现有参与者与SDV的轨迹将不同于原始日志。本文提出UniSim——一种神经传感器模拟器，它能将搭载传感器的车辆捕获的单一记录日志转换为逼真的闭环多传感器模拟。UniSim构建神经特征网格以重建场景中的静态背景与动态交通参与者，并通过合成二者来模拟新视角下的激光雷达和摄像头数据，支持交通参与者的增删与位置重设。为更好处理外推视角，我们为动态对象引入可学习先验，并利用卷积网络补全未观测区域。实验表明，UniSim在下游任务中能够以较小的域差距模拟逼真的传感器数据。借助UniSim，我们如同在真实世界中一样，在安全关键场景上对自主系统进行了闭环评估。