Code-trained language models have proven to be highly effective for various code intelligence tasks. However, they can be challenging to train and deploy for many software engineering applications due to computational bottlenecks and memory constraints. Implementing effective strategies to address these issues requires a better understanding of these 'black box' models. In this paper, we perform the first neuron-level analysis for source code models to identify \textit{important} neurons within latent representations. We achieve this by eliminating neurons that are highly similar or irrelevant to the given task. This approach helps us understand which neurons and layers can be eliminated (redundancy analysis) and where important code properties are located within the network (concept analysis). Using redundancy analysis, we make observations relevant to knowledge transfer and model optimization applications. We find that over 95\% of the neurons are redundant with respect to our code intelligence tasks and can be eliminated without significant loss in accuracy. We also discover several subsets of neurons that can make predictions with baseline accuracy. Through concept analysis, we explore the traceability and distribution of human-recognizable concepts within latent code representations which could be used to influence model predictions. We trace individual and subsets of important neurons to specific code properties and identify 'number' neurons, 'string' neurons, and higher-level 'text' neurons for token-level tasks and higher-level concepts important for sentence-level downstream tasks. This also helps us understand how decomposable and transferable task-related features are and can help devise better techniques for transfer learning, model compression, and the decomposition of deep neural networks into modules.

翻译：代码训练语言模型已被证明在多种代码智能任务中极为有效。然而，由于计算瓶颈和内存限制，这些模型在许多软件工程应用中的训练和部署仍面临挑战。为制定有效的应对策略，需更深入地理解这些"黑箱"模型。本文首次对源代码模型进行神经元级分析，以识别潜在表征中的**重要**神经元。我们通过消除与给定任务高度相似或无关的神经元实现这一目标。该方法有助于理解哪些神经元和层可以被消除（冗余分析），以及网络中重要代码属性的定位（概念分析）。通过冗余分析，我们获得了与知识迁移和模型优化应用相关的观察结果：超过95%的神经元在代码智能任务中是冗余的，可在不显著损失准确率的情况下被消除；同时发现若干神经元子集能以基线准确率进行预测。通过概念分析，我们探索了潜在代码表征中人类可识别概念的可追溯性与分布——这些概念可用于影响模型预测。我们将单个及神经元子集追溯至特定代码属性，识别出面向词元级任务的"数字"神经元、"字符串"神经元和高级"文本"神经元，以及面向句子级下游任务的重要高层概念。这有助于理解任务相关特征的可分解性与可迁移性，并为迁移学习、模型压缩及深度神经网络模块化分解提供更优技术方案。