We present VideoFactory, an innovative framework for generating high-quality open-domain videos. VideoFactory excels in producing high-definition (1376x768), widescreen (16:9) videos without watermarks, creating an engaging user experience. Generating videos guided by text instructions poses significant challenges, such as modeling the complex relationship between space and time, and the lack of large-scale text-video paired data. Previous approaches extend pretrained text-to-image generation models by adding temporal 1D convolution/attention modules for video generation. However, these approaches overlook the importance of jointly modeling space and time, inevitably leading to temporal distortions and misalignment between texts and videos. In this paper, we propose a novel approach that strengthens the interaction between spatial and temporal perceptions. In particular, we utilize a swapped cross-attention mechanism in 3D windows that alternates the "query" role between spatial and temporal blocks, enabling mutual reinforcement for each other. To fully unlock model capabilities for high-quality video generation, we curate a large-scale video dataset called HD-VG-130M. This dataset comprises 130 million text-video pairs from the open-domain, ensuring high-definition, widescreen and watermark-free characters. Objective metrics and user studies demonstrate the superiority of our approach in terms of per-frame quality, temporal correlation, and text-video alignment, with clear margins.

翻译：我们提出VideoFactory，一个用于生成高质量开放域视频的创新框架。VideoFactory擅长生成高清（1376×768）、宽屏（16：9）无水印视频，为用户带来沉浸式体验。通过文本指令引导视频生成面临显著挑战，例如对空间与时间复杂关系的建模，以及缺乏大规模文本-视频配对数据。现有方法通过添加时序1D卷积/注意力模块扩展预训练的文本-图像生成模型以实现视频生成，但这些方法忽视了空间与时间联合建模的重要性，不可避免地导致时序扭曲及文本与视频间的错配。本文提出一种创新方法，增强空间感知与时间感知间的交互。具体而言，我们在3D窗口中应用交换交叉注意力机制，使空间块与时间块交替承担"查询"角色，实现两者的相互强化。为充分释放模型的高质量视频生成能力，我们构建了名为HD-VG-130M的大规模视频数据集。该数据集包含来自开放域的1.3亿文本-视频对，确保高清、宽屏且无水印的视频内容。客观指标与用户研究均表明，本方法在帧质量、时序相关性及文本-视频对齐方面具有显著优势。