Placing a human in the loop may abate the risks of deploying AI systems in safety-critical settings (e.g., a clinician working with a medical AI system). However, mitigating risks arising from human error and uncertainty within such human-AI interactions is an important and understudied issue. In this work, we study human uncertainty in the context of concept-based models, a family of AI systems that enable human feedback via concept interventions where an expert intervenes on human-interpretable concepts relevant to the task. Prior work in this space often assumes that humans are oracles who are always certain and correct. Yet, real-world decision-making by humans is prone to occasional mistakes and uncertainty. We study how existing concept-based models deal with uncertain interventions from humans using two novel datasets: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset, and CUB-S, a relabeling of the popular CUB concept dataset with rich, densely-annotated soft labels from humans. We show that training with uncertain concept labels may help mitigate weaknesses of concept-based systems when handling uncertain interventions. These results allow us to identify several open challenges, which we argue can be tackled through future multidisciplinary research on building interactive uncertainty-aware systems. To facilitate further research, we release a new elicitation platform, UElic, to collect uncertain feedback from humans in collaborative prediction tasks.

翻译：将人类纳入决策循环可能降低在安全关键场景（如临床医生与医疗AI系统协作）部署AI系统的风险。然而，在此类人机交互过程中，由人为错误与不确定性引发的风险缓解策略仍是一个重要但研究不足的问题。本研究聚焦概念模型（基于概念的AI系统家族，通过概念干预机制使专家能够介入与任务相关且可解释的人工概念）中的人类不确定性。此前相关研究常假设人类是始终确定且正确的"先知"，但现实世界中的人类决策难免存在偶然性错误与不确定性。我们通过两个全新数据集探究现有概念模型如何处理带有不确定性的人类干预：UMNIST（基于MNIST数据集构建、具有受控模拟不确定性的视觉数据集）和CUB-S（对经典CUB概念数据集进行重新标注，包含丰富且密集标注的人类软标签）。研究表明，使用不确定性概念标签进行训练有助于弥补概念系统在处理不确定性干预时的缺陷。基于这些发现，我们识别出若干开放性挑战，并主张通过构建交互式不确定性感知系统的跨学科研究加以解决。为促进后续研究，我们发布了新型数据采集平台UElic，用于在协作预测任务中收集人类的不确定性反馈。