The arrival of Sora marks a new era for text-to-video diffusion models, bringing significant advancements in video generation and potential applications. However, Sora, along with other text-to-video diffusion models, is highly reliant on prompts, and there is no publicly available dataset that features a study of text-to-video prompts. In this paper, we introduce VidProM, the first large-scale dataset comprising 1.67 Million unique text-to-Video Prompts from real users. Additionally, this dataset includes 6.69 million videos generated by four state-of-the-art diffusion models, alongside some related data. We initially discuss the curation of this large-scale dataset, a process that is both time-consuming and costly. Subsequently, we underscore the need for a new prompt dataset specifically designed for text-to-video generation by illustrating how VidProM differs from DiffusionDB, a large-scale prompt-gallery dataset for image generation. Our extensive and diverse dataset also opens up many exciting new research areas. For instance, we suggest exploring text-to-video prompt engineering, efficient video generation, and video copy detection for diffusion models to develop better, more efficient, and safer models. The project (including the collected dataset VidProM and related code) is publicly available at https://vidprom.github.io under the CC-BY-NC 4.0 License.

翻译：Sora的到来标志着文本到视频扩散模型进入新时代，在视频生成及潜在应用方面带来显著进展。然而，Sora与其他文本到视频扩散模型高度依赖提示词，且目前尚无公开数据集可用于文本到视频提示词的研究。本文提出VidProM——首个包含来自真实用户的167万条独特文本到视频提示词的大规模数据集。此外，该数据集包含由四种最先进的扩散模型生成的669万个视频及相关辅助数据。我们首先讨论了这一大规模数据集的构建过程（此过程耗时且成本高昂），随后通过阐明VidProM与面向图像生成的大规模提示画廊数据集DiffusionDB之间的差异，强调了构建专门用于文本到视频生成的提示数据集的必要性。我们广泛且多样化的数据集还开辟了许多激动人心的新研究方向，例如建议探索文本到视频提示工程、高效视频生成及扩散模型视频拷贝检测，以开发更优、更高效且更安全的模型。该项目（包括所收集的VidProM数据集及相关代码）在CC-BY-NC 4.0许可协议下公开于https://vidprom.github.io。