As artificial intelligence plays an increasingly substantial role in decisions affecting humans and society, the accountability of automated decision systems has been receiving increasing attention from researchers and practitioners. Fairness, which is concerned with eliminating unjust treatment and discrimination against individuals or sensitive groups, is a critical aspect of accountability. Yet, for evaluating fairness, there is a plethora of fairness metrics in the literature that employ different perspectives and assumptions that are often incompatible. This work focuses on group fairness. Most group fairness metrics desire a parity between selected statistics computed from confusion matrices belonging to different sensitive groups. Generalizing this intuition, this paper proposes a new equal confusion fairness test to check an automated decision system for fairness and a new confusion parity error to quantify the extent of any unfairness. To further analyze the source of potential unfairness, an appropriate post hoc analysis methodology is also presented. The usefulness of the test, metric, and post hoc analysis is demonstrated via a case study on the controversial case of COMPAS, an automated decision system employed in the US to assist judges with assessing recidivism risks. Overall, the methods and metrics provided here may assess automated decision systems' fairness as part of a more extensive accountability assessment, such as those based on the system accountability benchmark.

翻译：随着人工智能在影响人类社会的决策中扮演日益重要的角色，自动决策系统的问责性正受到研究人员和实践者越来越多关注。公平性作为消除对个人或敏感群体不公正对待与歧视的关键方面，是问责性的核心要素。然而，文献中存在大量采用不同视角和假设的公平性度量标准，这些标准往往互不兼容。本文聚焦于群体公平性。大多数群体公平性度量标准期望不同敏感群体混淆矩阵中计算的选定统计量之间达到均等。基于这一直觉，本文提出了一种新的等困惑公平性检验方法用于检测自动决策系统的公平性，以及一种新的困惑均等误差用于量化任何不公平的程度。为深入分析潜在不公平的来源，还提出了相应的后验分析方法。通过对美国用于辅助法官评估再犯风险的自动决策系统COMPAS争议案例的实证研究，展示了该检验、度量及后验分析的有效性。总体而言，本文提供的方法和度量可作为更广泛问责性评估（如基于系统问责性基准的评估）的一部分，用于评估自动决策系统的公平性。