Measuring bias is key for better understanding and addressing unfairness in NLP/ML models. This is often done via fairness metrics which quantify the differences in a model's behaviour across a range of demographic groups. In this work, we shed more light on the differences and similarities between the fairness metrics used in NLP. First, we unify a broad range of existing metrics under three generalized fairness metrics, revealing the connections between them. Next, we carry out an extensive empirical comparison of existing metrics and demonstrate that the observed differences in bias measurement can be systematically explained via differences in parameter choices for our generalized metrics.

翻译：衡量偏见是更好地认识和解决NLP/ML模型中不公平现象的关键,这往往是通过公平度量标准来做到的,这种公平度量标准可以量化模式在一系列人口群体中的行为差异。在这项工作中,我们更清楚地说明国家LP中使用的公平度量标准之间的差异和相似性。首先,我们将现有的各种计量标准统一在三种普遍公平度标准之下,揭示它们之间的联系。接着,我们对现有的计量标准进行广泛的经验比较,并表明通过我们通用度量参数选择的差异,可以系统地解释观察到的偏见度量差异。