Subjective ratings contain inherent noise that limits the model-human correlation, but this reliability issue is rarely quantified. In this paper, we present $ρ$-Perfect, a practical estimation of the highest achievable correlation of a model on subjectively rated datasets. We define $ρ$-Perfect to be the correlation between a perfect predictor and human ratings, and derive an estimate of the value based on heteroscedastic noise scenarios, a common occurrence in subjectively rated datasets. We show that $ρ$-Perfect squared estimates test-retest correlation and use this to validate the estimate. We demonstrate the use of $ρ$-Perfect on a speech quality dataset and show how the measure can distinguish between model limitations and data quality issues.

翻译：主观评分包含固有噪声，这会限制模型与人类之间的相关性，但这种可靠性问题很少被量化。本文提出ρ-Perfect，这是一种对模型在主观评分数据集上可达到的最高相关性的实用估计。我们将ρ-Perfect定义为完美预测器与人类评分之间的相关性，并基于异方差噪声场景（主观评分数据集中常见的情况）推导出该值的估计。我们证明ρ-Perfect的平方可估计重测相关性，并利用这一点验证该估计。我们在一个语音质量数据集上演示了ρ-Perfect的应用，并展示了该度量如何区分模型限制与数据质量问题。