A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits the rapid and efficient progress of language generators. One exception is the recently proposed Mauve. In theory, Mauve measures an information-theoretic divergence between two probability distributions over strings: one representing the language generator under evaluation; the other representing the true natural language distribution. Mauve's authors argue that its success comes from the qualitative properties of their proposed divergence. Yet in practice, as this divergence is uncomputable, Mauve approximates it by measuring the divergence between multinomial distributions over clusters instead, where cluster assignments are attained by grouping strings based on a pre-trained language model's embeddings. As we show, however, this is not a tight approximation -- in either theory or practice. This begs the question: why does Mauve work so well? In this work, we show that Mauve was right for the wrong reasons, and that its newly proposed divergence is not necessary for its high performance. In fact, classical divergences paired with its proposed cluster-based approximation may actually serve as better evaluation metrics. We finish the paper with a probing analysis; this analysis leads us to conclude that -- by encoding syntactic- and coherence-level features of text, while ignoring surface-level features -- such cluster-based substitutes to string distributions may simply be better for evaluating state-of-the-art language generators.

翻译：一个好的自动评估指标用于语言生成时，理想情况下应与人类对文本质量的判断高度相关。然而，此类指标的匮乏阻碍了语言生成器的快速高效进展。最近提出的Mauve是一个例外。理论上，Mauve测量两个字符串概率分布之间的信息论散度：一个代表被评估的语言生成器，另一个代表真实自然语言分布。Mauve的作者认为其成功源于所提出的散度的定性性质。然而，在实践中，由于此散度不可计算，Mauve通过测量聚类上多项式分布之间的散度来近似它，其中聚类分配是通过基于预训练语言模型的嵌入对字符串分组实现的。但我们表明，这在理论和实践中都不是紧近似。这引出一个问题：为什么Mauve效果如此之好？在本工作中，我们证明Mauve正确的原因有误，其新提出的散度并非高性能所必需。事实上，经典散度结合其提出的基于聚类的近似可能反而是更好的评估指标。我们以探测分析结束本文；该分析使我们得出结论——通过编码文本的句法和一致性层次特征而忽略表层特征——这种基于聚类的字符串分布替代可能更适用于评估最先进的语言生成器。