Biased human decisions have consequential impacts across various domains, yielding unfair treatment of individuals and resulting in suboptimal outcomes for organizations and society. In recognition of this fact, organizations regularly design and deploy interventions aimed at mitigating these biases. However, measuring human decision biases remains an important but elusive task. Organizations are frequently concerned with mistaken decisions disproportionately affecting one group. In practice, however, this is typically not possible to assess due to the scarcity of a gold standard: a label that indicates what the correct decision would have been. In this work, we propose a machine learning-based framework to assess bias in human-generated decisions when gold standard labels are scarce. We provide theoretical guarantees and empirical evidence demonstrating the superiority of our method over existing alternatives. This proposed methodology establishes a foundation for transparency in human decision-making, carrying substantial implications for managerial duties, and offering potential for alleviating algorithmic biases when human decisions are used as labels to train algorithms.

翻译：人类决策中的偏见在各领域均产生深远影响，既导致对个体的不公平对待，也为组织和社会带来次优结果。认识到这一事实后，各类组织经常设计并实施旨在缓解这些偏见的干预措施。然而，对人类决策偏见的测量始终是一项重要却难以实现的任务。组织通常关注对特定群体造成不成比例影响的错误决策，但在实践中，由于缺乏"黄金标准"——即指示正确决策应为何种的标签——这种评估往往无法实现。本研究提出一种基于机器学习的框架，用于在黄金标准标签稀缺的情况下评估人类决策中的偏见。我们提供了理论保证和实证证据，证明该方法优于现有替代方案。所提出的方法论为人类决策透明度奠定了基础，对管理职责具有重要影响，并在人类决策被用作算法训练标签时，为缓解算法偏见提供了潜在可能。