In response to everyday queries, humans explicitly signal uncertainty and offer alternative answers when they are unsure. Machine learning models that output calibrated prediction sets through conformal prediction mimic this human behaviour; larger sets signal greater uncertainty while providing alternatives. In this work, we study the usefulness of conformal prediction sets as an aid for human decision making by conducting a pre-registered randomized controlled trial with conformal prediction sets provided to human subjects. With statistical significance, we find that when humans are given conformal prediction sets their accuracy on tasks improves compared to fixed-size prediction sets with the same coverage guarantee. The results show that quantifying model uncertainty with conformal prediction is helpful for human-in-the-loop decision making and human-AI teams.

翻译：针对日常查询，人类会在不确定时明确表达不确定并提供替代答案。通过共形预测输出校准后预测集的机器学习模型模仿了这种行为：更大的预测集既体现了更高的不确定性，也提供了替代方案。本研究通过一项预先注册的随机对照试验，向人类受试者提供共形预测集，探究共形预测集作为辅助工具对人类决策的效用。具有统计学意义的结果表明，与具有相同覆盖保证的固定大小预测集相比，人类在获得共形预测集后其任务准确性得到提升。这些结果证明，利用共形预测量化模型不确定性对于含人类反馈的决策过程及人机协作团队具有实际价值。