The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services are released for generating high-visual quality videos. However, these methods often use a few academic metrics, for example, FVD or IS, to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a new framework and pipeline to exhaustively evaluate the performance of the generated videos. To achieve this, we first conduct a new prompt list for text-to-video generation by analyzing the real-world prompt list with the help of the large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmarks, in terms of visual qualities, content qualities, motion qualities, and text-caption alignment with around 18 objective metrics. To obtain the final leaderboard of the models, we also fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed opinion alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.

翻译：近年来，视觉与语言生成模型发展迅猛。在视频生成领域，各类开源模型及公开可用服务已发布，用于生成高质量视频。然而，这些方法通常仅使用少数学术指标（如FVD或IS）来评估性能。我们认为，由于这些模型通常在超大规模数据集上训练且具备多方面能力，仅凭简单指标难以评判大型条件生成模型。为此，我们提出一套全新的框架与流程，以全面评估生成视频的性能。为实现该目标，我们首先借助大语言模型分析真实世界提示词列表，构建了面向文本到视频生成的新提示词列表。随后，我们基于精心设计的基准，从视觉质量、内容质量、动态质量及文本-字幕对齐四个维度，使用约18项客观指标评估当前最先进的视频生成模型。为构建最终模型排行榜，我们还拟合了一系列系数，将客观指标与用户意见对齐。基于所提意见对齐方法，最终评分相比于简单平均指标呈现更高相关性，验证了所提评估方法的有效性。