When we use algorithms to produce risk assessments, we typically think of these predictions as providing helpful input to human decisions, such as when risk scores are presented to judges or doctors. But when a decision-maker obtains algorithmic assistance, they may not only react to the information. The decision-maker may view the input of the algorithm as recommending a default action, making it costly for them to deviate, such as when a judge is reluctant to overrule a high-risk assessment of a defendant or a doctor fears the consequences of deviating from recommended procedures. In this article, we propose a principal-agent model of joint human-machine decision-making. Within this model, we consider the effect and design of algorithmic recommendations when they affect choices not just by shifting beliefs, but also by altering preferences. We motivate this assumption from institutional factors, such as a desire to avoid audits, as well as from well-established models in behavioral science that predict loss aversion relative to a reference point, which here is set by the algorithm. We show that recommendation-dependent preferences create inefficiencies where the decision-maker is overly responsive to the recommendation. As a potential remedy, we discuss algorithms that strategically withhold recommendations, and show how they can improve the quality of final decisions.

翻译：当我们使用算法产生风险评估时，通常认为这些预测能为人类决策提供有益参考，例如向法官或医生呈现风险评分。但当决策者获得算法辅助时，他们可能不仅对信息本身做出反应。决策者可能将算法输入视为默认行动建议，从而使其偏离该建议产生成本——例如法官不愿推翻对被告的高风险评估，或医生担心偏离推荐程序可能带来的后果。本文提出一个联合人机决策的委托-代理模型。在该模型中，我们考察算法建议的影响与设计，这些建议不仅通过改变信念影响选择，还通过改变偏好影响行为。我们从制度因素（如规避审查的意愿）以及行为科学中预测参照点依赖型损失厌恶的成熟模型（此处的参照点由算法设定）出发，论证这一假设的合理性。研究表明，推荐依赖型偏好会导致决策者对推荐反应过度，产生效率损失。作为潜在解决方案，我们探讨了策略性地保留推荐的算法，并展示其如何提升最终决策质量。