In response to everyday queries, humans explicitly signal uncertainty and offer alternative answers when they are unsure. Machine learning models that output calibrated prediction sets through conformal prediction mimic this human behaviour; larger sets signal greater uncertainty while providing alternatives. In this work, we study the usefulness of conformal prediction sets as an aid for human decision making by conducting a pre-registered randomized controlled trial with conformal prediction sets provided to human subjects. With statistical significance, we find that when humans are given conformal prediction sets their accuracy on tasks improves compared to fixed-size prediction sets with the same coverage guarantee. The results show that quantifying model uncertainty with conformal prediction is helpful for human-in-the-loop decision making and human-AI teams.

翻译：针对日常查询，人类在不确定时会明确表达犹豫并提供替代答案。通过共形预测输出校准预测集的机器学习模型模拟了这种人类行为——更大的预测集既表征更高不确定性，又提供了替代选择。本研究通过开展预先注册的随机对照试验，向人类受试者提供共形预测集，探究此类预测集作为人类决策辅助工具的有效性。在统计显著性层面，我们发现：相较于具有相同覆盖保证的固定规模预测集，人类在获得共形预测集后任务准确率显著提升。结果表明，将共形预测用于量化模型不确定性，对人机协作决策具有实际助益。