We present MicroCinema, a straightforward yet effective framework for high-quality and coherent text-to-video generation. Unlike existing approaches that align text prompts with video directly, MicroCinema introduces a Divide-and-Conquer strategy which divides the text-to-video into a two-stage process: text-to-image generation and image\&text-to-video generation. This strategy offers two significant advantages. a) It allows us to take full advantage of the recent advances in text-to-image models, such as Stable Diffusion, Midjourney, and DALLE, to generate photorealistic and highly detailed images. b) Leveraging the generated image, the model can allocate less focus to fine-grained appearance details, prioritizing the efficient learning of motion dynamics. To implement this strategy effectively, we introduce two core designs. First, we propose the Appearance Injection Network, enhancing the preservation of the appearance of the given image. Second, we introduce the Appearance Noise Prior, a novel mechanism aimed at maintaining the capabilities of pre-trained 2D diffusion models. These design elements empower MicroCinema to generate high-quality videos with precise motion, guided by the provided text prompts. Extensive experiments demonstrate the superiority of the proposed framework. Concretely, MicroCinema achieves SOTA zero-shot FVD of 342.86 on UCF-101 and 377.40 on MSR-VTT. See https://wangyanhui666.github.io/MicroCinema.github.io/ for video samples.

翻译：我们提出MicroCinema，一个简单而高效的框架，用于生成高质量且连贯的文本到视频内容。与现有直接将文本提示与视频对齐的方法不同，MicroCinema采用“分而治之”策略，将文本到视频任务分解为两个阶段：文本到图像生成与图像及文本到视频生成。该策略具有两个显著优势：(a) 它使我们能够充分利用文本到图像模型（如Stable Diffusion、Midjourney和DALLE）的最新进展，生成照片级逼真且高度细节化的图像；(b) 借助生成的图像，模型可减少对细粒度外观细节的关注，优先高效学习运动动态。为实现该策略，我们提出两项核心设计。首先，提出外观注入网络，以增强对给定图像外观的保留能力。其次，引入外观噪声先验——一种旨在维持预训练二维扩散模型能力的新机制。这些设计要素使MicroCinema能够根据提供的文本提示，生成具有精确运动的高质量视频。大量实验证明了所提框架的优越性。具体而言，MicroCinema在UCF-101数据集上实现了342.86的零样本FVD最优结果，在MSR-VTT上达到了377.40。视频样本请见https://wangyanhui666.github.io/MicroCinema.github.io/。