The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services are released for generating high-visual quality videos. However, these methods often use a few academic metrics, for example, FVD or IS, to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a new framework and pipeline to exhaustively evaluate the performance of the generated videos. To achieve this, we first conduct a new prompt list for text-to-video generation by analyzing the real-world prompt list with the help of the large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmarks, in terms of visual qualities, content qualities, motion qualities, and text-caption alignment with around 18 objective metrics. To obtain the final leaderboard of the models, we also fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed opinion alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.

翻译：近年来，视觉与语言生成模型发展迅速。在视频生成领域，各类开源模型及公开服务被广泛发布，用于生成高视觉质量的视频。然而，这些方法常采用少量学术指标（例如FVD或IS）评估性能。我们认为，仅通过简单指标难以评判大型条件生成模型，因为这些模型通常基于海量数据集训练，具备多方面的生成能力。为此，我们提出了一种新的框架与流程，用于全面评估生成视频的性能。具体而言，首先借助大语言模型分析真实世界提示列表，构建了一个面向文本到视频生成的新提示列表。随后，基于我们精心设计的基准测试，从视觉质量、内容质量、运动质量及文本描述对齐四个方面，采用约18项客观指标对当前最先进的视频生成模型进行评测。为生成模型的最终排行榜，我们还拟合了一系列系数，使客观指标与用户意见对齐。基于所提出的意见对齐方法，我们得到的最终评分相较于简单平均指标具有更高的相关性，验证了该评估方法的有效性。