Trust calibration between humans and Artificial Intelligence (AI) is crucial for optimal decision-making in collaborative settings. Excessive trust can lead users to accept AI-generated outputs without question, overlooking critical flaws, while insufficient trust may result in disregarding valuable insights from AI systems, hindering performance. Despite its importance, there is currently no definitive and objective method for measuring trust calibration between humans and AI. Current approaches lack standardization and consistent metrics that can be broadly applied across various contexts, and they don't distinguish between the formation of opinions and subsequent human decisions. In this work, we propose a novel and objective method for dynamic trust calibration, introducing a standardized trust calibration measure and an indicator. By utilizing Contextual Bandits-an adaptive algorithm that incorporates context into decision-making-our indicator dynamically assesses when to trust AI contributions based on learned contextual information. We evaluate this indicator across three diverse datasets, demonstrating that effective trust calibration results in significant improvements in decision-making performance, as evidenced by 10 to 38% increase in reward metrics. These findings not only enhance theoretical understanding but also provide practical guidance for developing more trustworthy AI systems supporting decisions in critical domains, for example, disease diagnoses and criminal justice.

翻译：人类与人工智能（AI）之间的信任校准对于协作环境中的最优决策至关重要。过度信任可能导致用户不加质疑地接受AI生成的输出，忽视关键缺陷；而信任不足则可能导致忽略AI系统提供的宝贵见解，从而阻碍性能提升。尽管信任校准至关重要，但目前尚无明确且客观的方法来测量人类与AI之间的信任校准。现有方法缺乏标准化和可广泛应用于不同场景的一致度量标准，并且未能区分意见形成与后续人类决策。本研究提出了一种新颖且客观的动态信任校准方法，引入了一种标准化的信任校准度量及指示器。通过利用上下文赌博机——一种将上下文信息纳入决策的自适应算法——我们的指示器能够基于学习到的上下文信息动态评估何时信任AI的贡献。我们在三个不同的数据集上对该指示器进行了评估，结果表明有效的信任校准能显著提升决策性能，奖励指标提高了10%至38%。这些发现不仅增强了理论理解，还为开发更可信的AI系统提供了实践指导，以支持关键领域（例如疾病诊断和刑事司法）的决策。