Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g., ROUGE) and embedding-based measures (e.g., BERTScore), often misjudge semantic similarity of documents. Our study shows that both token-overlap metrics and embedding-based metrics routinely assign nearly identical scores to texts that directly contradict each other, thereby potentially masking fundamental errors. We introduce MATCHA, an automatic metric that jointly rewards semantic agreement with a reference and penalizes contradictions. MATCHA employs a dual-view perspective that measures (i) proximity to the gold text and (ii) distance from an adversarially generated counterfactual contradiction. In eight public benchmarks, MATCHA outperforms popular metrics, compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks. On the TruthfulQA dataset (i.e., a dataset without a training set, where no embedding-based metrics could locally train on), this improvement in terms of matching texts with a reference reaches 18.38% over ROUGE-L and 20.82% over BERTScore. Both quantitative comparison and qualitative human assessments confirm the efficacy and validity of MATCHA and uncover fundamental weaknesses in pre-existing metrics. Compared with 23 embedding models, including top state-of-the-art ones, used as a metric similar to BERTScore, MATCHA remains the most accurate in distinguishing correct from incorrect statements solely based on a reference. Our code and metric are publicly available (https://github.com/Siran-Li/MATCHA).

翻译：摘要：可靠的评估对于理解大语言模型（LLM）的性能至关重要，但当前的主流评价指标——即词元重叠分数（如ROUGE）和基于嵌入的度量（如BERTScore）——常常误判文档的语义相似性。本研究表明，词元重叠指标和基于嵌入的指标均会系统性地为直接相互矛盾的文本分配几乎相同的分数，从而可能掩盖根本性错误。我们提出MATCHA，一种自动评估指标，它同时奖励与参考文本的语义一致性并惩罚矛盾。MATCHA采用双视角方法，同时衡量：（i）与黄金文本的接近度，以及（ii）与对抗生成的反事实矛盾文本的距离。在八个公开基准测试中，MATCHA在问答、图像描述生成、自然语言推理、摘要生成和语义文本相似性任务上均优于主流指标，与人类标注的一致性更高。在TruthfulQA数据集（即无训练集的数据集，任何基于嵌入的指标均无法局部训练）上，MATCHA在文本与参考匹配方面的改进幅度较ROUGE-L提升18.38%，较BERTScore提升20.82%。定量比较与定性人工评估均证实了MATCHA的有效性与可靠性，并揭示了既有指标的根本性缺陷。与包括顶尖模型在内的23种嵌入模型（作为类似BERTScore的指标使用）相比，MATCHA在仅基于参考文本区分正确与错误陈述方面仍保持最高准确率。我们的代码与评估指标已公开（https://github.com/Siran-Li/MATCHA）。