Explainable artificial intelligence techniques are developed at breakneck speed, but suitable evaluation approaches lag behind. With explainers becoming increasingly complex and a lack of consensus on how to assess their utility, it is challenging to judge the benefit and effectiveness of different explanations. To address this gap, we take a step back from sophisticated predictive algorithms and instead look into explainability of simple decision-making models. In this setting, we aim to assess how people perceive comprehensibility of their different representations such as mathematical formulation, graphical representation and textual summarisation (of varying complexity and scope). This allows us to capture how diverse stakeholders -- engineers, researchers, consumers, regulators and the like -- judge intelligibility of fundamental concepts that more elaborate artificial intelligence explanations are built from. This position paper charts our approach to establishing appropriate evaluation methodology as well as a conceptual and practical framework to facilitate setting up and executing relevant user studies.

翻译：可解释人工智能技术以惊人的速度发展，但合适的评估方法却相对滞后。随着解释器日益复杂，且缺乏评估其效用的共识，判断不同解释的益处和有效性成为挑战。为弥补这一差距，我们从复杂的预测算法退一步，转而审视简单决策模型的可解释性。在此背景下，我们旨在评估人们如何感知不同表示形式（如数学公式、图形表示和文本摘要，这些形式具有不同的复杂性和范围）的可理解性。这使我们能够捕捉不同的利益相关者——工程师、研究人员、消费者、监管者等——如何判断构成更复杂人工智能解释基础的基本概念的可理解性。本文探讨了建立适当评估方法的方法，以及一个概念性和实践性框架，以促进设置和执行相关的用户研究。