World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.

翻译：世界模型在自动驾驶领域展现出显著优势，特别是在多视角驾驶视频生成方面。然而，生成定制化驾驶视频仍面临重大挑战。本文提出DriveDreamer-2，该方法在DriveDreamer框架基础上引入大语言模型（LLM），以生成用户定义的驾驶视频。具体而言，首先集成LLM接口将用户查询转化为智能体轨迹；随后基于轨迹生成遵循交通规则的高精地图（HDMap）；最终提出统一多视角模型以增强生成驾驶视频的时空连贯性。DriveDreamer-2是首个能够生成定制化驾驶视频的世界模型，能以用户友好方式生成罕见驾驶场景（如车辆突然切入）。实验结果表明，该模型生成的视频可有效提升驾驶感知方法（如3D检测与跟踪）的训练效果。此外，DriveDreamer-2的视频生成质量超越其他最先进方法，FID和FVD分数分别达到11.2和55.7，相对提升30%和50%。