One of the motivations for explainable AI is to allow humans to make better and more informed decisions regarding the use and deployment of AI models. But careful evaluations are needed to assess whether this expectation has been fulfilled. Current evaluations mainly focus on algorithmic properties of explanations, and those that involve human subjects often employ subjective questions to test human's perception of explanation usefulness, without being grounded in objective metrics and measurements. In this work, we evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. We conduct a mixed-methods user study involving image data to evaluate saliency maps generated by SmoothGrad, GradCAM, and an oracle explanation on two tasks: model selection and counterfactual simulation. To our surprise, we did not find evidence of significant improvement on these tasks when users were provided with any of the saliency maps, even the synthetic oracle explanation designed to be simple to understand and highly indicative of the answer. Nonetheless, explanations did help users more accurately describe the models. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.

翻译：可解释人工智能的动机之一是使人类能够更明智地做出关于AI模型使用和部署的决策。然而，需要仔细评估这一期望是否已实现。目前的评估主要关注解释的算法特性，而涉及人类受试者的研究常采用主观问题测试人类对解释有用性的感知，未能基于客观指标和测量。在本研究中，我们评估了解释能否在机器学习模型开发的实际场景中改善人类决策。我们采用混合方法用户研究，以图像数据为基础，评估SmoothGrad、GradCAM及一个合成最优解释生成的显著性图在模型选择与反事实模拟两项任务中的表现。令人惊讶的是，我们发现即使采用设计得简单易懂且高度指示答案的合成最优解释，用户在获得任何显著性图后，也未能显著改进这两项任务的表现。尽管如此，解释确实帮助用户更准确地描述模型。这些发现提示我们需谨慎看待基于显著性的解释的有用性及其可能引发的误解。