In response to everyday queries, humans explicitly signal uncertainty and offer alternative answers when they are unsure. Machine learning models that output calibrated prediction sets through conformal prediction mimic this human behaviour; larger sets signal greater uncertainty while providing alternatives. In this work, we study the usefulness of conformal prediction sets as an aid for human decision making by conducting a pre-registered randomized controlled trial with conformal prediction sets provided to human subjects. With statistical significance, we find that when humans are given conformal prediction sets their accuracy on tasks improves compared to fixed-size prediction sets with the same coverage guarantee. The results show that quantifying model uncertainty with conformal prediction is helpful for human-in-the-loop decision making and human-AI teams.

翻译：针对日常查询，人类在不确定时会明确表达不确定性并提供替代答案。通过共形预测输出校准预测集的机器学习模型能够模仿这种人类行为——更大的预测集在提供替代方案的同时也反映了更高的不确定性。本研究通过预注册随机对照试验，向人类受试者提供共形预测集，探究其作为人类决策辅助工具的有效性。具有统计显著性结果表明，相较于具有相同覆盖保证的固定大小预测集，当人类获得共形预测集时，其任务准确率得到提升。这些发现表明，利用共形预测量化模型不确定性有助于人在回路决策和人机协作。