We examine how users perceive the limitations of an AI system when it encounters a task that it cannot perform perfectly and whether providing explanations alongside its answers aids users in constructing an appropriate mental model of the system's capabilities and limitations. We employ a visual question answer and explanation task where we control the AI system's limitations by manipulating the visual inputs: during inference, the system either processes full-color or grayscale images. Our goal is to determine whether participants can perceive the limitations of the system. We hypothesize that explanations will make limited AI capabilities more transparent to users. However, our results show that explanations do not have this effect. Instead of allowing users to more accurately assess the limitations of the AI system, explanations generally increase users' perceptions of the system's competence - regardless of its actual performance.

翻译：本研究探讨当人工智能系统遇到无法完美执行的任务时，用户如何感知其局限性，以及提供解释是否有助于用户构建对系统能力与局限性的准确心智模型。我们采用视觉问答与解释任务，通过操控视觉输入（推理阶段系统处理彩色或灰度图像）来控制AI系统的局限性。研究旨在确定参与者能否感知系统的局限性。我们假设解释将使受限的AI能力对用户更加透明。然而，结果显示解释并未产生此效果。解释非但未能让用户更准确地评估AI系统的局限性，反而普遍提升了用户对系统能力的评价——无论其实际性能如何。