In this study, we analyze NLG automatic metrics based on whether human evaluation aspect is used as context or objective to compute the metrics: (i) Task-agnostic and (ii) Human-aligned. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remains unclear. We present metric preference checklist as a framework to assess the discriminative power of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. We show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly when a disagreement between human evaluation aspects is present. We also show particular use cases in which automatic metrics provide a better guidance than human on discriminating system-level performance. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless their correlation level to human; and (ii) for scrutinizing the strengths and limitations of NLG systems, which are often obscured by a standard averaging method of evaluation scores.

翻译：本研究基于评价指标是否将人类评估维度作为上下文或目标来计算，对NLG自动指标进行分析：（i）任务无关指标和（ii）人类对齐指标。任务无关指标（如困惑度、BLEU、BERTScore）成本低廉且高度适应多种NLG任务，但与人类判断的相关性较弱。人类对齐指标（CTC、CtrlEval、UniEval）通过将理想的人类特质作为训练目标来提升相关性水平，然而，它们在区分系统级性能和输出质量方面的有效性仍不明确。我们提出指标偏好检查表作为评估框架，检验自动指标在文本摘要、对话回复生成和可控生成三个NLG任务中的区分能力。研究表明，多维度人类对齐指标（UniEval）未必优于单维度人类对齐指标（CTC、CtrlEval）和任务无关指标（BLEU、BERTScore），尤其在人类评估维度存在分歧时。我们还展示了特定用例中自动指标在区分系统级性能方面比人类判断更具指导性。我们提出的框架可实现：（i）验证自动指标是否忠实反映人类偏好，无论其与人类判断的相关性水平如何；（ii）深入剖析NLG系统的优势与局限——这些特点常因标准化的评价分数平均方法而被掩盖。