Contemporary models for generating images show remarkable quality and versatility. Swayed by these advantages, the research community repurposes them to generate videos. Since video content is highly redundant, we argue that naively bringing advances of image models to the video generation domain reduces motion fidelity, visual quality and impairs scalability. In this work, we build Snap Video, a video-first model that systematically addresses these challenges. To do that, we first extend the EDM framework to take into account spatially and temporally redundant pixels and naturally support video generation. Second, we show that a U-Net - a workhorse behind image generation - scales poorly when generating videos, requiring significant computational overhead. Hence, we propose a new transformer-based architecture that trains 3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us to efficiently train a text-to-video model with billions of parameters for the first time, reach state-of-the-art results on a number of benchmarks, and generate videos with substantially higher quality, temporal consistency, and motion complexity. The user studies showed that our model was favored by a large margin over the most recent methods. See our website at https://snap-research.github.io/snapvideo/.

翻译：当前图像生成模型展现出卓越的质量与多用途性。受此优势影响，研究界尝试将其改造用于视频生成。由于视频内容存在高度冗余，我们认为直接将图像模型的进步草率应用于视频生成领域，会降低运动保真度与视觉质量，并损害可扩展性。在本研究中，我们构建了Snap Video——一种以视频为中心的系统性解决上述挑战的模型。为此，我们首先扩展EDM框架以考虑时空冗余像素，并天然支持视频生成。其次，我们证明U-Net（图像生成领域的核心架构）在视频生成时扩展性差，需消耗大量计算资源。因此，我们提出一种基于Transformer的新架构，其训练速度比U-Net快3.31倍（推理速度约快4.5倍）。这使得我们首次高效训练出拥有数十亿参数的文本到视频模型，在多项基准测试中达到最先进水平，并生成具有显著更高质量、时间一致性与运动复杂度的视频。用户研究表明，我们的模型以极大优势获得青睐。详情请访问官网：https://snap-research.github.io/snapvideo/。