Explainable AI was born as a pathway to allow humans to explore and understand the inner working of complex systems. However, establishing what is an explanation and objectively evaluating explainability are not trivial tasks. This paper presents a new model-agnostic metric to measure the Degree of Explainability of information in an objective way. We exploit a specific theoretical model from Ordinary Language Philosophy called the Achinstein's Theory of Explanations, implemented with an algorithm relying on deep language models for knowledge graph extraction and information retrieval. To understand whether this metric can measure explainability, we devised a few experiments and user studies involving more than 190 participants, evaluating two realistic systems for healthcare and finance using famous AI technology, including Artificial Neural Networks and TreeSHAP. The results we obtained are statistically significant (with P values lower than .01), suggesting that our proposed metric for measuring the Degree of Explainability is robust in several scenarios, and it aligns with concrete expectations.

翻译：可解释人工智能作为探索和理解复杂系统内部运作机制的途径应运而生。然而，界定何为解释以及客观评价可解释性并非易事。本文提出一种新的模型无关度量标准，用于客观衡量信息的可解释性程度。我们借鉴日常语言哲学中的特定理论模型——阿钦斯坦解释理论，通过结合知识图谱抽取与信息检索的深度语言模型算法加以实现。为验证该度量能否有效衡量可解释性，我们设计包含190余名参与者的实验与用户研究，针对医疗和金融两大实际应用场景，采用包括人工神经网络和TreeSHAP在内的知名AI技术进行评估。所得结果具有统计学显著性（P值低于0.01），表明我们所提出的可解释性程度度量标准在多种场景下具有鲁棒性，且与具体预期相吻合。