The arrival of Sora marks a new era for text-to-video diffusion models, bringing significant advancements in video generation and potential applications. However, Sora, along with other text-to-video diffusion models, is highly reliant on prompts, and there is no publicly available dataset that features a study of text-to-video prompts. In this paper, we introduce VidProM, the first large-scale dataset comprising 1.67 Million unique text-to-Video Prompts from real users. Additionally, this dataset includes 6.69 million videos generated by four state-of-the-art diffusion models, alongside some related data. We initially discuss the curation of this large-scale dataset, a process that is both time-consuming and costly. Subsequently, we underscore the need for a new prompt dataset specifically designed for text-to-video generation by illustrating how VidProM differs from DiffusionDB, a large-scale prompt-gallery dataset for image generation. Our extensive and diverse dataset also opens up many exciting new research areas. For instance, we suggest exploring text-to-video prompt engineering, efficient video generation, and video copy detection for diffusion models to develop better, more efficient, and safer models. The project (including the collected dataset VidProM and related code) is publicly available at https://vidprom.github.io under the CC-BY-NC 4.0 License.

翻译：Sora的出现标志着文本到视频扩散模型新时代的到来，为视频生成及其潜在应用带来了重大进展。然而，Sora及其他文本到视频扩散模型高度依赖提示词，但目前尚无公开数据集支持对文本到视频提示词的系统性研究。本文提出VidProM——首个包含来自真实用户的167万条独特文本到视频提示词的大规模数据集。此外，该数据集还包含由四种最先进扩散模型生成的669万个视频及相关数据。我们首先讨论了这一耗时且成本高昂的大规模数据集构建过程，随后通过对比图像生成领域的大规模提示库数据集DiffusionDB，阐明了针对文本到视频生成专门设计的新型提示数据集的必要性。本数据集覆盖面广、多样性强，为众多新兴研究领域开辟了道路。例如，我们建议探索文本到视频提示工程、高效视频生成以及面向扩散模型的视频副本检测，以开发更优、更高效且更安全的模型。该项目（包含收集的数据集VidProM及相关代码）已通过CC-BY-NC 4.0许可协议在https://vidprom.github.io公开提供。