World models, especially in autonomous driving, are trending and drawing extensive attention due to their capacity for comprehending driving environments. The established world model holds immense potential for the generation of high-quality driving videos, and driving policies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gaming environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model entirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the powerful diffusion model to construct a comprehensive representation of the complex environment. Furthermore, we introduce a two-stage training pipeline. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. The proposed DriveDreamer is the first world model established from real-world driving scenarios. We instantiate DriveDreamer on the challenging nuScenes benchmark, and extensive experiments verify that DriveDreamer empowers precise, controllable video generation that faithfully captures the structural constraints of real-world traffic scenarios. Additionally, DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.

翻译：世界模型，特别是在自动驾驶领域，因其理解驾驶环境的能力而备受关注并引发广泛研究。已建立的世界模型在生成高质量驾驶视频及安全驾驶策略方面具有巨大潜力。然而，相关研究的一个关键局限在于其主要聚焦于游戏环境或仿真场景，缺乏对真实世界驾驶情境的表征。为此，我们提出DriveDreamer，一种完全源自真实驾驶场景的开创性世界模型。鉴于复杂驾驶场景中世界建模面临巨大的搜索空间，我们提出利用强大的扩散模型构建复杂环境的综合表征。此外，我们引入两阶段训练流程：初始阶段使DriveDreamer深入理解结构化交通约束，后续阶段赋予其预测未来状态的能力。所提出的DriveDreamer是首个基于真实驾驶场景建立的世界模型。我们在具有挑战性的nuScenes基准上实例化DriveDreamer，大量实验验证了DriveDreamer能够实现精确、可控的视频生成，并忠实捕获真实交通场景的结构约束。同时，DriveDreamer能够生成真实且合理的驾驶策略，为交互与实际应用开辟了新途径。