Recent literature has seen a significant focus on building machine learning models with specific properties such as fairness, i.e., being non-biased with respect to a given set of attributes, calibration i.e., model confidence being aligned with its predictive accuracy, and explainability, i.e., ability to be understandable to humans. While there has been work focusing on each of these aspects individually, researchers have shied away from simultaneously addressing more than one of these dimensions. In this work, we address the problem of building models which are both fair and calibrated. We work with a specific definition of fairness, which closely matches [Biswas et. al. 2019], and has the nice property that Bayes optimal classifier has the maximum possible fairness under our definition. We show that an existing negative result towards achieving a fair and calibrated model [Kleinberg et. al. 2017] does not hold for our definition of fairness. Further, we show that ensuring group-wise calibration with respect to the sensitive attributes automatically results in a fair model under our definition. Using this result, we provide a first cut approach for achieving fair and calibrated models, via a simple post-processing technique based on temperature scaling. We then propose modifications of existing calibration losses to perform group-wise calibration, as a way of achieving fair and calibrated models in a variety of settings. Finally, we perform extensive experimentation of these techniques on a diverse benchmark of datasets, and present insights on the pareto-optimality of the resulting solutions.

翻译：近期文献重点关注构建具有特定属性的机器学习模型，例如公平性（即对给定属性集合无偏）、校准性（即模型置信度与其预测精度一致）以及可解释性（即易于人类理解）。尽管已有研究分别探讨这些方面，但研究者尚未同时解决多个维度的问题。本文针对构建兼具公平性与校准性的模型展开研究。我们采用与 [Biswas et al. 2019] 高度一致的公平性定义，其优良性质在于：贝叶斯最优分类器在该定义下可实现最大公平性。我们证明，现有关于实现公平且校准模型的负面结论 [Kleinberg et al. 2017] 不适用于本文的公平性定义。进一步地，我们论证针对敏感属性进行分组校准可自动得到符合该定义的公平模型。基于此结果，我们提出一种基于温度缩放的简单后处理方法，作为实现公平且校准模型的初步方案。随后，我们通过修改现有校准损失函数实现分组校准，从而在多种场景下构建公平且校准的模型。最后，我们在多样化基准数据集上对这些方法进行广泛实验，并深入分析所得解的帕累托最优性。