Despite the progress we have recorded in scaling multilingual machine translation (MT) models and evaluation data to several under-resourced African languages, it is difficult to measure accurately the progress we have made on these languages because evaluation is often performed on n-gram matching metrics like BLEU that often have worse correlation with human judgments. Embedding-based metrics such as COMET correlate better; however, lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with a simplified MQM guideline for error-span annotation and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET, a COMET evaluation metric for African languages by leveraging DA training data from high-resource languages and African-centric multilingual encoder (AfroXLM-Roberta) to create the state-of-the-art evaluation metric for African languages MT with respect to Spearman-rank correlation with human judgments (+0.406).

翻译：尽管我们在将多语言机器翻译（MT）模型和评估数据扩展到多种资源匮乏的非洲语言方面取得了进展，但由于评估通常采用与人工判断相关性较差的n元语法匹配指标（如BLEU），我们很难准确衡量在这些语言上所取得的进步。基于嵌入的指标（如COMET）相关性更好；然而，资源匮乏语言缺乏带人工评分评估数据、多维质量指标（MQM）等标注指南的复杂性、以及多语言编码器有限的语言覆盖范围，阻碍了这些指标在非洲语言上的应用。本文中，我们通过以下方式应对这些挑战：采用简化的MQM指南，为13种类型多样的非洲语言创建了高质量的人工评估数据，涵盖错误跨度标注和直接评估（DA）评分。此外，我们开发了AfriCOMET——一种面向非洲语言的COMET评估指标，通过利用高资源语言的DA训练数据和以非洲为中心的多语言编码器（AfroXLM-Roberta），创建了在斯皮尔曼等级相关性方面与人工判断达到最优水平（+0.406）的非洲语言机器翻译评估指标。