We present MicroCinema, a straightforward yet effective framework for high-quality and coherent text-to-video generation. Unlike existing approaches that align text prompts with video directly, MicroCinema introduces a Divide-and-Conquer strategy which divides the text-to-video into a two-stage process: text-to-image generation and image\&text-to-video generation. This strategy offers two significant advantages. a) It allows us to take full advantage of the recent advances in text-to-image models, such as Stable Diffusion, Midjourney, and DALLE, to generate photorealistic and highly detailed images. b) Leveraging the generated image, the model can allocate less focus to fine-grained appearance details, prioritizing the efficient learning of motion dynamics. To implement this strategy effectively, we introduce two core designs. First, we propose the Appearance Injection Network, enhancing the preservation of the appearance of the given image. Second, we introduce the Appearance Noise Prior, a novel mechanism aimed at maintaining the capabilities of pre-trained 2D diffusion models. These design elements empower MicroCinema to generate high-quality videos with precise motion, guided by the provided text prompts. Extensive experiments demonstrate the superiority of the proposed framework. Concretely, MicroCinema achieves SOTA zero-shot FVD of 342.86 on UCF-101 and 377.40 on MSR-VTT. See https://wangyanhui666.github.io/MicroCinema.github.io/ for video samples.

翻译：我们提出MicroCinema，一种简洁而高效的框架，用于生成高质量且连贯的文本到视频内容。与现有方法直接将文本提示与视频对齐不同，MicroCinema引入了一种分治策略，将文本到视频生成分解为两个阶段：文本到图像生成和图像与文本到视频生成。该策略具有两大优势：a) 它使我们能够充分利用文本到图像模型的最新进展（如Stable Diffusion、Midjourney和DALLE），生成逼真且细节丰富的图像；b) 借助生成的图像，模型可以减少对细粒度外观细节的关注，优先高效学习运动动力学。为了有效实施该策略，我们引入了两项核心设计。首先，我们提出外观注入网络，增强对给定图像外观的保持能力。其次，我们引入外观噪声先验，一种旨在维持预训练二维扩散模型能力的新机制。这些设计元素使MicroCinema能够根据提供的文本提示生成具有精确运动的高质量视频。大量实验证明了所提框架的优越性。具体而言，MicroCinema在UCF-101和MSR-VTT数据集上分别实现了342.86和377.40的零样本FVD最优性能。视频样本见https://wangyanhui666.github.io/MicroCinema.github.io/。