Traditional algorithmic fairness notions rely on label feedback, which can only be elicited from expert critics. However, in most practical applications, several non-expert stakeholders also play a major role in the system and can have distinctive opinions about the decision making philosophy. For example, in kidney placement programs, transplant surgeons are very wary about accepting kidney offers for black patients due to genetic reasons. However, non-expert stakeholders in kidney placement programs (e.g. patients, donors and their family members) may misinterpret such decisions from the perspective of social discrimination. This paper evaluates group fairness notions from the viewpoint of non-expert stakeholders, who can only provide binary \emph{agreement/disagreement feedback} regarding the decision in context. Specifically, two types of group fairness notions have been identified: (i) \emph{definite notions} (e.g. calibration), which can be evaluated exactly using disagreement feedback, and (ii) \emph{indefinite notions} (e.g. equal opportunity) which suffer from uncertainty due to lack of label feedback. In the case of indefinite notions, bounds are presented based on disagreement rates, and an estimate is constructed based on established bounds. The efficacy of all our findings are validated empirically on real human feedback dataset.

翻译：传统算法公平概念依赖于标签反馈，而这种反馈通常只能从专家批评者处获取。然而，在大多数实际应用中，若干非专家利益相关者也在系统中扮演重要角色，且可能对决策理念持有独特见解。例如，在肾脏分配项目中，移植外科医生因遗传因素对接受黑人患者的肾脏捐赠持审慎态度。但该项目中的非专家利益相关者（如患者、捐赠者及其家属）可能从社会歧视角度误读此类决策。本文从非专家利益相关者的视角评估群体公平概念——这类群体仅能针对具体情境下的决策提供二元"同意/不同意"反馈。具体而言，研究识别出两类群体公平概念：（i）确定性概念（如校准性），可利用异议反馈精确评估；及（ii）非确定性概念（如机会均等），因缺乏标签反馈而存在不确定性。针对非确定性概念，本文基于异议率给出了界限范围，并依据既有界限构建了估计量。所有研究结果均在真实人类反馈数据集上进行了实证验证。