We present VideoFactory, an innovative framework for generating high-quality open-domain videos. VideoFactory excels in producing high-definition (1376x768), widescreen (16:9) videos without watermarks, creating an engaging user experience. Generating videos guided by text instructions poses significant challenges, such as modeling the complex relationship between space and time, and the lack of large-scale text-video paired data. Previous approaches extend pretrained text-to-image generation models by adding temporal 1D convolution/attention modules for video generation. However, these approaches overlook the importance of jointly modeling space and time, inevitably leading to temporal distortions and misalignment between texts and videos. In this paper, we propose a novel approach that strengthens the interaction between spatial and temporal perceptions. In particular, we utilize a swapped cross-attention mechanism in 3D windows that alternates the "query" role between spatial and temporal blocks, enabling mutual reinforcement for each other. To fully unlock model capabilities for high-quality video generation, we curate a large-scale video dataset called HD-VG-130M. This dataset comprises 130 million text-video pairs from the open-domain, ensuring high-definition, widescreen and watermark-free characters. Objective metrics and user studies demonstrate the superiority of our approach in terms of per-frame quality, temporal correlation, and text-video alignment, with clear margins.

翻译：本文提出VideoFactory，一种用于生成高质量开放域视频的创新框架。VideoFactory擅长生成无水印的高清（1376x768）、宽屏（16:9）视频，为用户带来沉浸式体验。基于文本指令生成视频面临显著挑战，例如建模时空复杂关系，以及缺乏大规模文本-视频配对数据。现有方法通过添加一维时序卷积/注意力模块扩展预训练的文本-图像生成模型。然而，这些方法忽略了联合建模时空的重要性，不可避免地导致时序扭曲以及文本与视频间的失配。本文提出一种增强时空感知交互的新方法：具体而言，我们在三维窗口中采用交换交叉注意力机制，交替赋予时空模块"查询"角色，实现二者的相互增强。为充分释放模型的高质量视频生成能力，我们构建了名为HD-VG-130M的大规模视频数据集。该数据集包含1.3亿个开放域文本-视频配对，确保高清、宽屏且无文字水印的特征。客观指标与用户研究均表明，本方法在逐帧质量、时序关联性及文本-视频对齐方面具有显著优势。