Rigorous testing of autonomous robots, such as self-driving vehicles, is essential to ensure their safety in real-world deployments. This requires building high-fidelity simulators to test scenarios beyond those that can be safely or exhaustively collected in the real-world. Existing neural rendering methods based on NeRF and 3DGS hold promise but suffer from low rendering speeds or can only render pinhole camera models, hindering their suitability to applications that commonly require high-distortion lenses and LiDAR data. Multi-sensor simulation poses additional challenges as existing methods handle cross-sensor inconsistencies by favoring the quality of one modality at the expense of others. To overcome these limitations, we propose SimULi, the first method capable of rendering arbitrary camera models and LiDAR data in real-time. Our method extends 3DGUT, which natively supports complex camera models, with LiDAR support, via an automated tiling strategy for arbitrary spinning LiDAR models and ray-based culling. To address cross-sensor inconsistencies, we design a factorized 3D Gaussian representation and anchoring strategy that reduces mean camera and depth error by up to 40% compared to existing methods. SimULi renders 10-20x faster than ray tracing approaches and 1.5-10x faster than prior rasterization-based work (and handles a wider range of camera models). When evaluated on two widely benchmarked autonomous driving datasets, SimULi matches or exceeds the fidelity of existing state-of-the-art methods across numerous camera and LiDAR metrics.

翻译：自动驾驶车辆等自主机器人的严格测试对于确保其在真实场景中的安全性至关重要。这需要构建高保真模拟器，以测试那些无法在现实世界中安全或全面收集的场景。现有基于NeRF和3DGS的神经渲染方法虽然展现出潜力，但存在渲染速度慢或仅能渲染针孔相机模型的局限，使其难以适用于通常需要高畸变镜头和激光雷达数据的应用场景。多传感器模拟面临更多挑战，因为现有方法在处理跨传感器不一致性时，往往以牺牲某一模态质量为代价来提升另一模态的效果。为克服这些局限，我们提出SimULi——首个能够实时渲染任意相机模型与激光雷达数据的方法。我们的方法扩展了原生支持复杂相机模型的3DGUT，通过针对任意旋转式激光雷达模型的自动分块策略和基于射线的裁剪技术为其添加激光雷达支持。为解决跨传感器不一致性问题，我们设计了分解式三维高斯表征和锚定策略，相比现有方法可将平均相机误差和深度误差降低高达40%。SimULi的渲染速度比光线追踪方法快10-20倍，比先前的光栅化方法快1.5-10倍（并支持更广泛的相机模型）。在两个广泛基准测试的自动驾驶数据集上评估时，SimULi在多项相机和激光雷达指标上达到或超越了现有最先进方法的保真度。