Generating video stories from text prompts is a complex task. In addition to having high visual quality, videos need to realistically adhere to a sequence of text prompts whilst being consistent throughout the frames. Creating a benchmark for video generation requires data annotated over time, which contrasts with the single caption used often in video datasets. To fill this gap, we collect comprehensive human annotations on three existing datasets, and introduce StoryBench: a new, challenging multi-task benchmark to reliably evaluate forthcoming text-to-video models. Our benchmark includes three video generation tasks of increasing difficulty: action execution, where the next action must be generated starting from a conditioning video; story continuation, where a sequence of actions must be executed starting from a conditioning video; and story generation, where a video must be generated from only text prompts. We evaluate small yet strong text-to-video baselines, and show the benefits of training on story-like data algorithmically generated from existing video captions. Finally, we establish guidelines for human evaluation of video stories, and reaffirm the need of better automatic metrics for video generation. StoryBench aims at encouraging future research efforts in this exciting new area.

翻译：从文本提示生成视频故事是一项复杂任务。视频不仅需要具备高视觉质量，还需真实地遵循一系列文本提示序列，并保持帧间一致性。创建视频生成基准需要随时间标注的数据，这与视频数据集中常用的单条描述截然不同。为填补这一空白，我们在三个现有数据集上收集了全面的人工标注，并引入StoryBench：一个新颖、具有挑战性的多任务基准，用于可靠评估即将出现的文本到视频模型。该基准包含三个难度递增的视频生成任务：动作执行——需从条件视频出发生成下一个动作；故事延续——需从条件视频出发执行一系列动作；故事生成——仅依据文本提示生成视频。我们评估了规模较小但性能强劲的文本到视频基线模型，并展示了通过现有视频描述算法生成的故事类数据进行训练的优势。最后，我们制定了视频故事的人工评估准则，并重申了视频生成领域需要更优自动评估指标的迫切性。StoryBench旨在鼓励这一令人兴奋的新兴领域中的未来研究工作。