The use of Artificial Intelligence (AI) based on data-driven algorithms has become ubiquitous in today's society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions as compared to a human alone or AI an alone. We introduce a new methodological framework that can be used to answer experimentally this question with no additional assumptions. We measure a decision maker's ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded experimental design, in which the provision of AI-generated recommendations is randomized across cases with a human making final decisions. Under this experimental design, we show how to compare the performance of three alternative decision-making systems--human-alone, human-with-AI, and AI-alone. We apply the proposed methodology to the data from our own randomized controlled trial of a pretrial risk assessment instrument. We find that AI recommendations do not improve the classification accuracy of a judge's decision to impose cash bail. Our analysis also shows that AI-alone decisions generally perform worse than human decisions with or without AI assistance. Finally, AI recommendations tend to impose cash bail on non-white arrestees more often than necessary when compared to white arrestees.

翻译：基于数据驱动算法的人工智能（AI）已广泛应用于当今社会。然而在许多场景中，尤其是当决策结果至关重要时，最终决定仍由人类做出。因此，关键问题在于：与人类单独决策或AI单独决策相比，AI辅助是否能让人类做出更优决策？我们提出了一种无需额外假设即可通过实验回答该问题的新方法学框架。通过基线潜在结果的标准分类指标，我们评估决策者做出正确判断的能力。我们采用单盲实验设计，随机分配AI生成的建议是否提供给人类决策者，由人类做出最终裁决。在该实验设计下，我们展示了如何比较三种决策系统（人类单独决策、人类辅助AI决策、AI单独决策）的绩效。我们将所提出的方法论应用于自身开展的审前风险评估工具随机对照试验数据。研究发现：AI建议并未提高法官决定是否实施现金保释的分类准确率；同时，无论是有人类参与还是纯AI决策，AI单独决策的表现通常逊于人类决策。此外，与白人被捕者相比，AI建议对非裔被捕者更频繁地施加了不必要的现金保释。