Despite rapid advancements in TTS models, a consistent and robust human evaluation framework is still lacking. For example, MOS tests fail to differentiate between similar models, and CMOS's pairwise comparisons are time-intensive. The MUSHRA test is a promising alternative for evaluating multiple TTS systems simultaneously, but in this work we show that its reliance on matching human reference speech unduly penalises the scores of modern TTS systems that can exceed human speech quality. More specifically, we conduct a comprehensive assessment of the MUSHRA test, focusing on its sensitivity to factors such as rater variability, listener fatigue, and reference bias. Based on our extensive evaluation involving 492 human listeners across Hindi and Tamil we identify two primary shortcomings: (i) reference-matching bias, where raters are unduly influenced by the human reference, and (ii) judgement ambiguity, arising from a lack of clear fine-grained guidelines. To address these issues, we propose two refined variants of the MUSHRA test. The first variant enables fairer ratings for synthesized samples that surpass human reference quality. The second variant reduces ambiguity, as indicated by the relatively lower variance across raters. By combining these approaches, we achieve both more reliable and more fine-grained assessments. We also release MANGO, a massive dataset of 246,000 human ratings, the first-of-its-kind collection for Indian languages, aiding in analyzing human preferences and developing automatic metrics for evaluating TTS systems.

翻译：尽管文本转语音（TTS）模型发展迅速，但一致且稳健的人工评估框架仍然缺乏。例如，平均意见得分（MOS）测试难以区分相似模型，而比较平均意见得分（CMOS）的成对比较则耗时费力。MUSHRA测试作为同时评估多个TTS系统的潜在替代方案显示出前景，但本研究表明，其依赖匹配人类参考语音的做法会不公正地降低那些可能超越人类语音质量的现代TTS系统的得分。具体而言，我们对MUSHRA测试进行了全面评估，重点关注其对评分者变异性、听者疲劳和参考偏见等因素的敏感性。基于我们在印地语和泰米尔语中涉及492名听者的广泛评估，我们识别出两个主要缺陷：（i）参考匹配偏见，即评分者受到人类参考的不当影响；（ii）判断模糊性，源于缺乏清晰细粒度的指导原则。为解决这些问题，我们提出了两种改进的MUSHRA测试变体。第一种变体能够为合成质量超越人类参考的样本提供更公平的评分。第二种变体通过相对降低评分者间的方差来减少模糊性。结合这两种方法，我们实现了更可靠且更细粒度的评估。我们还发布了MANGO数据集，这是一个包含246,000条人工评分的大规模数据集，作为首个针对印度语言的此类集合，有助于分析人类偏好并开发用于评估TTS系统的自动指标。