We propose a new interpretable measure of unfairness, that allows providing a quantitative analysis of classifier fairness, beyond a dichotomous fair/unfair distinction. We show how this measure can be calculated when the classifier's conditional confusion matrices are known. We further propose methods for auditing classifiers for their fairness when the confusion matrices cannot be obtained or even estimated. Our approach lower-bounds the unfairness of a classifier based only on aggregate statistics, which may be provided by the owner of the classifier or collected from freely available data. We use the equalized odds criterion, which we generalize to the multiclass case. We report experiments on data sets representing diverse applications, which demonstrate the effectiveness and the wide range of possible uses of the proposed methodology. An implementation of the procedures proposed in this paper and as the code for running the experiments are provided in https://github.com/sivansabato/unfairness.

翻译：我们提出一种新的可解释的不公平度量方法，能够对分类器的公平性进行定量分析，而不仅仅是二元性的公平/不公平区分。我们展示了当分类器的条件混淆矩阵已知时，如何计算该度量。我们进一步提出在无法获取甚至无法估计混淆矩阵时对分类器公平性进行审计的方法。我们的方法仅基于汇总统计量即可给出分类器不公平性的下界，这些统计量可由分类器所有者提供或从自由可用的数据中收集。我们采用了均衡几率准则，并将其推广到多类情形。我们在代表不同应用场景的数据集上进行了实验，证明了所提方法的有效性及广泛适用性。本文所提程序及实验代码的实现见 https://github.com/sivansabato/unfairness。