In this paper, we present MovieFactory, a powerful framework to generate cinematic-picture (3072$\times$1280), film-style (multi-scene), and multi-modality (sounding) movies on the demand of natural languages. As the first fully automated movie generation model to the best of our knowledge, our approach empowers users to create captivating movies with smooth transitions using simple text inputs, surpassing existing methods that produce soundless videos limited to a single scene of modest quality. To facilitate this distinctive functionality, we leverage ChatGPT to expand user-provided text into detailed sequential scripts for movie generation. Then we bring scripts to life visually and acoustically through vision generation and audio retrieval. To generate videos, we extend the capabilities of a pretrained text-to-image diffusion model through a two-stage process. Firstly, we employ spatial finetuning to bridge the gap between the pretrained image model and the new video dataset. Subsequently, we introduce temporal learning to capture object motion. In terms of audio, we leverage sophisticated retrieval models to select and align audio elements that correspond to the plot and visual content of the movie. Extensive experiments demonstrate that our MovieFactory produces movies with realistic visuals, diverse scenes, and seamlessly fitting audio, offering users a novel and immersive experience. Generated samples can be found in YouTube or Bilibili (1080P).

翻译：本文提出MovieFactory，一个强大的框架，可根据自然语言需求生成电影级画面（3072×1280）、电影风格（多场景）和多模态（含配音）影片。据我们所知，作为首个全自动电影生成模型，该方法使用户仅通过简单文本输入即可创作具有平滑过渡的引人入胜的电影，超越了现有仅能生成单场景、无声且质量有限视频的方法。为实现这一独特功能，我们利用ChatGPT将用户提供的文本扩展为详细的电影生成脚本序列，随后通过视觉生成和音频检索将脚本在视觉和听觉上生动呈现。在视频生成方面，我们通过两阶段流程扩展预训练文本到图像扩散模型的能力：首先采用空间微调弥合预训练图像模型与新视频数据集之间的差异，随后引入时序学习捕捉物体运动。在音频方面，我们借助先进的检索模型选择并对齐与电影情节和视觉内容相匹配的音频元素。大量实验表明，MovieFactory能够生成具有逼真视觉效果、多样场景及完美契合音频的电影，为用户提供新颖且沉浸式的体验。生成的样本可在YouTube或Bilibili（1080P）获得。