Several recent works encourage the use of a Bayesian framework when assessing performance and fairness metrics of a classification algorithm in a supervised setting. We propose the Uncertainty Matters (UM) framework that generalizes a Beta-Binomial approach to derive the posterior distribution of any criteria combination, allowing stable performance assessment in a bias-aware setting.We suggest modeling the confusion matrix of each demographic group using a Multinomial distribution updated through a Bayesian procedure. We extend UM to be applicable under the popular K-fold cross-validation procedure. Experiments highlight the benefits of UM over classical evaluation frameworks regarding informativeness and stability.

翻译：近期多项研究鼓励在监督学习场景下评估分类算法的性能与公平性指标时采用贝叶斯框架。我们提出不确定性重要（Uncertainty Matters, UM）框架，该框架泛化了贝塔-二项分布方法，可推导任意指标组合的后验分布，从而在考虑偏差的背景下实现稳定的性能评估。我们建议使用多项分布对每个群体的混淆矩阵进行建模，并通过贝叶斯过程进行更新。我们将UM扩展至适用于流行的K折交叉验证流程。实验表明，与经典评估框架相比，UM在信息量及稳定性方面具有显著优势。