Rigorous testing of autonomous robots, such as self-driving vehicles, is essential to ensure their safety in real-world deployments. This requires building high-fidelity simulators to test scenarios beyond those that can be safely or exhaustively collected in the real-world. Existing neural rendering methods based on NeRF and 3DGS hold promise but suffer from low rendering speeds or can only render pinhole camera models, hindering their suitability to applications that commonly require high-distortion lenses and LiDAR data. Multi-sensor simulation poses additional challenges as existing methods handle cross-sensor inconsistencies by favoring the quality of one modality at the expense of others. To overcome these limitations, we propose SimULi, the first method capable of rendering arbitrary camera models and LiDAR data in real-time. Our method extends 3DGUT, which natively supports complex camera models, with LiDAR support, via an automated tiling strategy for arbitrary spinning LiDAR models and ray-based culling. To address cross-sensor inconsistencies, we design a factorized 3D Gaussian representation and anchoring strategy that reduces mean camera and depth error by up to 40% compared to existing methods. SimULi renders 10-20x faster than ray tracing approaches and 1.5-10x faster than prior rasterization-based work (and handles a wider range of camera models). When evaluated on two widely benchmarked autonomous driving datasets, SimULi matches or exceeds the fidelity of existing state-of-the-art methods across numerous camera and LiDAR metrics.

翻译：对自动驾驶车辆等自主机器人进行严格测试，对于确保其在实际部署中的安全性至关重要。这需要构建高保真仿真器，以测试那些在现实世界中无法安全或详尽采集的场景。现有的基于NeRF和3DGS的神经渲染方法虽前景广阔，但存在渲染速度低或仅能渲染针孔相机模型的问题，限制了其在通常需要高畸变镜头和激光雷达数据的应用中的适用性。多传感器仿真带来了额外的挑战，因为现有方法处理跨传感器不一致性时，往往以牺牲其他模态质量为代价来优先保证某一模态的质量。为克服这些局限，我们提出了SimULi，这是首个能够实时渲染任意相机模型和激光雷达数据的方法。我们的方法扩展了原生支持复杂相机模型的3DGUT，通过针对任意旋转激光雷达模型的自动分块策略和基于射线的剔除，为其增加了激光雷达支持。为解决跨传感器不一致性问题，我们设计了一种因子化的3D高斯表示与锚定策略，与现有方法相比，可将平均相机和深度误差降低高达40%。SimULi的渲染速度比光线追踪方法快10-20倍，比先前的基于光栅化的工作快1.5-10倍（并且支持更广泛的相机模型）。在两个广泛使用的自动驾驶基准数据集上进行评估时，SimULi在众多相机和激光雷达指标上达到或超越了现有最先进方法的保真度。