In this paper, we review recent approaches for explaining concepts in neural networks. Concepts can act as a natural link between learning and reasoning: once the concepts are identified that a neural learning system uses, one can integrate those concepts with a reasoning system for inference or use a reasoning system to act upon them to improve or enhance the learning system. On the other hand, knowledge can not only be extracted from neural networks but concept knowledge can also be inserted into neural network architectures. Since integrating learning and reasoning is at the core of neuro-symbolic AI, the insights gained from this survey can serve as an important step towards realizing neuro-symbolic AI based on explainable concepts.

翻译：本文综述了近年来用于解释神经网络概念的方法。概念可视为连接学习与推理的自然桥梁：一旦识别出神经学习系统所使用的概念，便可将这些概念与推理系统整合以执行推理，或利用推理系统对其进行操作来改进或增强学习系统。另一方面，知识不仅能从神经网络中提取，概念知识也可注入神经网络架构。由于学习与推理的整合是神经符号人工智能的核心，本综述所获得的洞见可成为迈向基于可解释概念的神经符号人工智能的重要一步。