Automated decision-making systems are becoming increasingly ubiquitous, motivating an immediate need for their explainability. However, it remains unclear whether users know what insights an explanation offers and, more importantly, what information it lacks. We conducted an online study with 200 participants to assess explainees' ability to realise known and unknown information for four representative explanations: transparent modelling, decision boundary visualisation, counterfactual explainability and feature importance. Our findings demonstrate that feature importance and decision boundary visualisation are the most comprehensible, but their limitations are not necessarily recognised by the users. In addition, correct interpretation of an explanation -- i.e., understanding known information -- is accompanied by high confidence, but a failure to gauge its limits -- thus grasp unknown information -- yields overconfidence; the latter phenomenon is especially prominent for feature importance and transparent modelling. Machine learning explanations should therefore embrace their richness and limitations to maximise understanding and curb misinterpretation.

翻译：自动化决策系统正变得日益普及，这迫切要求系统具备可解释性。然而，用户是否了解解释能提供哪些见解，更重要的是，能否意识到解释中缺失哪些信息，目前仍不明确。我们开展了一项包含200名参与者的在线研究，评估解释对象对四种代表性解释——透明建模、决策边界可视化、反事实可解释性和特征重要性——中已知和未知信息的认知能力。研究结果表明，特征重要性和决策边界可视化最易于理解，但用户未必能识别其局限性。此外，正确解读解释（即理解已知信息）伴随着高置信度，但未能评估其限度（从而掌握未知信息）会导致过度自信；后一种现象在特征重要性和透明建模中尤为突出。因此，机器学习解释应兼顾其丰富性与局限性，以最大化理解并减少误解。