Recently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on \textsc{webnlg}, Referring Expressions (REs) generated by the state-of-the-art neural models were not only indistinguishable from the REs in \textsc{webnlg} but also from the REs generated by a simple rule-based system. Here, we argue that this limitation could stem from the use of a purely ratings-based human evaluation (which is a common practice in Natural Language Generation). To investigate these issues, we propose an intrinsic task-based evaluation for REG models, in which, in addition to rating the quality of REs, participants were asked to accomplish two meta-level tasks. One of these tasks concerns the referential success of each RE; the other task asks participants to suggest a better alternative for each RE. The outcomes suggest that, in comparison to previous evaluations, the new evaluation protocol assesses the performance of each REG model more comprehensively and makes the participants' ratings more reliable and discriminable.

翻译：最近，一项关于指称表达生成（REG）模型的人类评估研究得出了意想不到的结论：在\textsc{webnlg}数据集上，最先进的神经模型生成的指称表达（REs）不仅与\textsc{webnlg}中的指称表达难以区分，而且与简单规则系统生成的指称表达也难以区分。在此，我们认为这一局限性可能源于纯基于评分的评估方式（这是自然语言生成的常见实践）。为探究这些问题，我们提出了一种面向REG模型的内在任务评估方法，该方法除了要求参与者对指称表达质量进行评分外，还要求他们完成两项元级任务。其中一项任务涉及每个指称表达的指称成功性；另一项任务要求参与者为每个指称表达提出更优的替代方案。结果表明，与以往评估相比，新评估协议能更全面地评估各REG模型的性能，并使参与者的评分更具可靠性和区分度。