In this study, we analyze NLG automatic metrics based on whether human evaluation aspect is used as context or objective to compute the metrics: (i) Task-agnostic and (ii) Human-aligned. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remains unclear. We present metric preference checklist as a framework to assess the discriminative power of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. We show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly when a disagreement between human evaluation aspects is present. We also show particular use cases in which automatic metrics provide a better guidance than human on discriminating system-level performance. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless their correlation level to human; and (ii) for scrutinizing the strengths and limitations of NLG systems, which are often obscured by a standard averaging method of evaluation scores.

翻译：本研究基于人类评估方面是否用作计算指标的上下文或目标，将NLG自动指标分为两类：（i）任务无关指标和（ii）人类对齐指标。任务无关指标（如困惑度、BLEU、BERTScore）具有成本效益高且适用于多种NLG任务的优点，但与人类判断的相关性较弱。人类对齐指标（CTC、CtrlEval、UniEval）通过将类似人类的质量特性纳入训练目标来提高相关性水平。然而，它们在区分系统级性能和输出质量方面的有效性仍不明确。我们提出了指标偏好清单作为评估自动指标在三个NLG任务（文本摘要、对话回复生成、受控生成）中判别力的框架。研究表明，多维度人类对齐指标（UniEval）不一定优于单维度人类对齐指标（CTC、CtrlEval）和任务无关指标（BLEU、BERTScore），尤其是在人类评估方面存在分歧的情况下。我们还展示了自动指标在区分系统级性能方面优于人类判断的特定用例。我们提出的框架可提供：（i）验证自动指标是否忠实于人类偏好（无论其与人类的相关性水平如何）的途径；（ii）审视NLG系统优缺点的途径，这些优缺点常被评估分数的标准平均方法所掩盖。