Photorealistic 3D vehicle models with high controllability are essential for autonomous driving simulation and data augmentation. While handcrafted CAD models provide flexible controllability, free CAD libraries often lack the high-quality materials necessary for photorealistic rendering. Conversely, reconstructed 3D models offer high-fidelity rendering but lack controllability. In this work, we introduce UrbanCAD, a framework that generates highly controllable and photorealistic 3D vehicle digital twins from a single urban image, leveraging a large collection of free 3D CAD models and handcrafted materials. To achieve this, we propose a novel pipeline that follows a retrieval-optimization manner, adapting to observational data while preserving fine-grained expert-designed priors for both geometry and material. This enables vehicles' realistic 360-degree rendering, background insertion, material transfer, relighting, and component manipulation. Furthermore, given multi-view background perspective and fisheye images, we approximate environment lighting using fisheye images and reconstruct the background with 3DGS, enabling the photorealistic insertion of optimized CAD models into rendered novel view backgrounds. Experimental results demonstrate that UrbanCAD outperforms baselines in terms of photorealism. Additionally, we show that various perception models maintain their accuracy when evaluated on UrbanCAD with in-distribution configurations but degrade when applied to realistic out-of-distribution data generated by our method. This suggests that UrbanCAD is a significant advancement in creating photorealistic, safety-critical driving scenarios for downstream applications.

翻译：具备高可控性的照片级真实感三维车辆模型对于自动驾驶仿真与数据增强至关重要。虽然手工制作的CAD模型提供了灵活的可控性，但免费CAD库通常缺乏照片级渲染所需的高质量材质。相反，重建的三维模型能实现高保真渲染，却缺乏可控性。本研究提出UrbanCAD框架，该框架利用大量免费三维CAD模型与手工制作材质，从单张城市图像生成高可控且具照片级真实感的三维车辆数字孪生体。为实现这一目标，我们设计了一种遵循检索-优化范式的新型流程，既能适应观测数据，又能保留专家设计的几何与材质细粒度先验。这使得车辆能够实现逼真的360度渲染、背景融合、材质迁移、重光照及部件操控。此外，基于多视角背景透视图像与鱼眼图像，我们利用鱼眼图像近似估算环境光照，并采用3DGS重建背景，从而将优化后的CAD模型以照片级真实感融入新视角渲染背景中。实验结果表明，UrbanCAD在照片真实感方面优于基线方法。同时，我们验证了多种感知模型在采用本分布配置的UrbanCAD数据上评估时能保持其准确性，但在应用于本方法生成的真实外分布数据时性能会出现下降。这表明UrbanCAD在创建面向下游应用的照片级真实感安全关键驾驶场景方面取得了重要进展。