Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, crucial for 3D perception tasks, remains elusive. Specifically, utilizing Bird's-Eye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, especially for 3D object detection tasks. In this paper, we introduce MagicDrive, a novel street view generation framework offering diverse 3D geometry controls, including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, our design incorporates a cross-view attention module, ensuring consistency across multiple camera views. With MagicDrive, we achieve high-fidelity street-view synthesis that captures nuanced 3D geometry and various scene descriptions, enhancing tasks like BEV segmentation and 3D object detection.

翻译：扩散模型的最新进展显著增强了具有二维控制的数据合成能力。然而，在街景生成中实现精确的三维控制（这对三维感知任务至关重要）仍然具有挑战性。具体而言，以鸟瞰图作为主要条件通常会导致几何控制（如高度）方面的难题，影响物体形状、遮挡模式及路面高程的表示，而这些对于感知数据合成（尤其是三维目标检测任务）均不可或缺。本文提出MagicDrive——一种新颖的街景生成框架，该框架通过定制化的编码策略，提供包括相机位姿、道路地图和三维边界框在内的多样化三维几何控制，并结合文本描述。此外，我们的设计集成了跨视图注意力模块，可确保多相机视角间的一致性。借助MagicDrive，我们实现了能够捕捉细微三维几何与多样化场景描述的高保真街景合成，从而提升了鸟瞰图分割与三维目标检测等任务的性能。