We present ART$\boldsymbol{\cdot}$V, an efficient framework for auto-regressive video generation with diffusion models. Unlike existing methods that generate entire videos in one-shot, ART$\boldsymbol{\cdot}$V generates a single frame at a time, conditioned on the previous ones. The framework offers three distinct advantages. First, it only learns simple continual motions between adjacent frames, therefore avoiding modeling complex long-range motions that require huge training data. Second, it preserves the high-fidelity generation ability of the pre-trained image diffusion models by making only minimal network modifications. Third, it can generate arbitrarily long videos conditioned on a variety of prompts such as text, image or their combinations, making it highly versatile and flexible. To combat the common drifting issue in AR models, we propose masked diffusion model which implicitly learns which information can be drawn from reference images rather than network predictions, in order to reduce the risk of generating inconsistent appearances that cause drifting. Moreover, we further enhance generation coherence by conditioning it on the initial frame, which typically contains minimal noise. This is particularly useful for long video generation. When trained for only two weeks on four GPUs, ART$\boldsymbol{\cdot}$V already can generate videos with natural motions, rich details and a high level of aesthetic quality. Besides, it enables various appealing applications, e.g., composing a long video from multiple text prompts.

翻译：我们提出了ART$\boldsymbol{\cdot}$V——一种基于扩散模型的高效自回归视频生成框架。与一次性生成完整视频的现有方法不同，ART$\boldsymbol{\cdot}$V逐帧生成，每帧基于前一帧的条件进行生成。该框架具有三大优势：首先，它仅需学习相邻帧间的简单连续运动，从而避免了需要大量训练数据建模的复杂长程运动；其次，通过仅对网络进行最小化修改，保留了预训练图像扩散模型的高保真生成能力；第三，它能基于文本、图像或两者组合等多种提示条件生成任意长度的视频，展现出高度通用性与灵活性。为解决自回归模型中常见的漂移问题，我们提出掩码扩散模型，该模型隐式学习从参考图像而非网络预测中提取信息，从而降低因生成不一致外观导致的漂移风险。此外，我们通过将生成过程约束在通常噪声极小的初始帧上，进一步增强生成连贯性——这对长视频生成尤为有效。在仅用四块GPU训练两周的情况下，ART$\boldsymbol{\cdot}$V即可生成具有自然运动、丰富细节与高审美质量的视频。同时，该框架还支持多种有趣应用，例如通过多个文本提示组合生成长视频。