Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.

翻译：近期，开放域文本到视频（T2V）生成模型取得了显著进展。然而，其令人瞩目的成果主要依赖于生成视频的定性案例展示，而T2V模型的定量评估仍面临两个关键问题。首先，现有研究缺乏对不同类别文本提示下T2V模型的细粒度评估。尽管部分基准已对提示进行分类，但其分类要么仅关注单一维度，要么未能考虑视频生成中的时序信息。其次，自动评估指标是否与人类判断标准一致尚不明确。为解决这些问题，我们提出FETV——面向文本到视频生成的细粒度评估基准。FETV具有多维度特性，基于三个正交维度对提示进行分类：主要内容、需控制的属性及提示复杂度。FETV同时具备时序感知能力，引入了专为视频生成设计的若干时序类别。基于FETV，我们对四种代表性T2V模型开展了全面的人工评估，揭示了它们在不同维度、不同类别提示上的优劣。我们还将FETV扩展为测试平台，用于评估自动T2V指标的可靠性。FETV的多维度分类体系可对指标在不同场景下的可靠性进行细粒度分析。研究发现，现有自动指标（如CLIPScore和FVD）与人工评估的相关性较弱。为解决该问题，我们探索了多种改进CLIPScore与FVD的方案，并开发出两种与人类判断相关性显著高于现有指标的自动评估指标。基准页面：https://github.com/llyx97/FETV。