In this paper, we review recent approaches for explaining concepts in neural networks. Concepts can act as a natural link between learning and reasoning: once the concepts are identified that a neural learning system uses, one can integrate those concepts with a reasoning system for inference or use a reasoning system to act upon them to improve or enhance the learning system. On the other hand, knowledge can not only be extracted from neural networks but concept knowledge can also be inserted into neural network architectures. Since integrating learning and reasoning is at the core of neuro-symbolic AI, the insights gained from this survey can serve as an important step towards realizing neuro-symbolic AI based on explainable concepts.

翻译：本文综述了近期在神经网络中解释概念的方法。概念可以充当学习与推理之间的天然纽带：一旦识别出神经学习系统所使用的概念，便可将这些概念与推理系统整合以进行推断，或利用推理系统对这些概念进行操作，以改进或增强学习系统。另一方面，知识不仅可以从神经网络中提取，概念知识也可以嵌入神经网络架构中。由于学习与推理的整合是神经符号人工智能的核心，本综述所获得的见解可为实现基于可解释概念的神经符号人工智能迈出重要一步。