The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train MantisScore (initialized from Mantis) based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman correlation between MantisScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and VBench show that MantisScore has consistently much higher correlation with human judges than other metrics. Due to these results, we believe MantisScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.

翻译：近年来，视频生成领域取得了巨大进展。然而，自动视频评估指标的发展却显著滞后。现有指标均无法对生成视频提供可靠的评分。主要障碍在于缺乏大规模的人工标注数据集。本文发布了VideoFeedback，这是首个大规模数据集，包含人类对来自11个现有视频生成模型的37.6K个合成视频提供的多维度评分。我们基于VideoFeedback训练了MantisScore（从Mantis初始化），以实现自动视频质量评估。实验表明，在VideoFeedback-test上，MantisScore与人类评分的斯皮尔曼相关系数可达77.1，比先前最佳指标高出约50分。在其他保留测试集EvalCrafter、GenAI-Bench和VBench上的进一步结果表明，与其他指标相比，MantisScore与人类评判者的相关性始终高出许多。基于这些结果，我们相信MantisScore可以作为人类评分者的优秀代理，以（1）评估不同视频模型以追踪进展；（2）在人类反馈强化学习（RLHF）中模拟细粒度的人类反馈，以改进当前的视频生成模型。