In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a ``coarse-to-fine'' process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames. This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap, and makes it possible to generate all segments in parallel. To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26\%) at the same hardware setting when generating 1024 frames. The homepage link is \url{https://msra-nuwa.azurewebsites.net/}

翻译：本文提出NUWA-XL，一种新颖的“扩散之上再扩散”架构，用于超长视频生成。当前大多数工作采用逐段顺序生成长视频的方式，这通常导致短视频训练与长视频推理之间的差距，且顺序生成效率低下。相反，我们的方法采用“从粗到细”的过程，使得视频能够以相同粒度并行生成。首先应用全局扩散模型生成整个时间范围内的关键帧，然后局部扩散模型递归地填充相邻帧之间的内容。这种简单而有效的策略使我们能够直接在长视频（3376帧）上训练，以减少训练-推理差距，并实现所有片段并行生成。为评估模型，我们构建了FlintstonesHD数据集，这是一个用于长视频生成的新基准。实验表明，我们的模型不仅能生成长度高质量（兼具全局与局部连贯性）的长视频，而且在相同硬件配置下生成1024帧时，将平均推理时间从7.55分钟降至26秒（降低94.26%）。主页链接：\url{https://msra-nuwa.azurewebsites.net/}