Recent developments in transformer-based language models have allowed them to capture a wide variety of world knowledge that can be adapted to downstream tasks with limited resources. However, what pieces of information are understood in these models is unclear, and neuron-level contributions in identifying them are largely unknown. Conventional approaches in neuron explainability either depend on a finite set of pre-defined descriptors or require manual annotations for training a secondary model that can then explain the neurons of the primary model. In this paper, we take BERT as an example and we try to remove these constraints and propose a novel and scalable framework that ties textual descriptions to neurons. We leverage the potential of generative language models to discover human-interpretable descriptors present in a dataset and use an unsupervised approach to explain neurons with these descriptors. Through various qualitative and quantitative analyses, we demonstrate the effectiveness of this framework in generating useful data-specific descriptors with little human involvement in identifying the neurons that encode these descriptors. In particular, our experiment shows that the proposed approach achieves 75% precision@2, and 50% recall@2

翻译：基于Transformer的语言模型近期发展使其能够捕获广泛的世界知识，并可适配到资源有限的各类下游任务中。然而，这些模型具体理解哪些信息尚不明确，而神经元层面在识别这些信息中的贡献也大多未知。传统的神经元可解释性方法要么依赖于一组有限的预定义描述符，要么需要手动标注以训练一个能解释主模型神经元的辅助模型。本文以BERT为例，尝试突破这些限制，提出一种新颖且可扩展的框架，将文本描述与神经元绑定。我们利用生成式语言模型的潜力来发现数据集中可被人类理解的描述符，并通过无监督方法解释这些描述符对应的神经元。通过多种定性与定量分析，我们验证了该框架在很少需要人工干预的情况下生成特定数据描述符的有效性，以及识别编码这些描述符的神经元能力。特别地，实验结果表明，所提方法在精确率@2上达到75%，召回率@2上达到50%。