Neural code intelligence models continue to be 'black boxes' to the human programmer. This opacity limits their application towards code intelligence tasks, particularly for applications like vulnerability detection where a model's reliance on spurious correlations can be safety-critical. We introduce a neuron-level approach to interpretability of neural code intelligence models which eliminates redundancy due to highly similar or task-irrelevant neurons within these networks. We evaluate the remaining important neurons using probing classifiers which are often used to ascertain whether certain properties have been encoded within the latent representations of neural intelligence models. However, probing accuracies may be artificially inflated due to repetitive and deterministic nature of tokens in code datasets. Therefore, we adapt the selectivity metric originally introduced in NLP to account for probe memorization, to formulate our source-code probing tasks. Through our neuron analysis, we find that more than 95\% of the neurons are redundant wrt. our code intelligence tasks and can be eliminated without significant loss in accuracy. We further trace individual and subsets of important neurons to specific code properties which could be used to influence model predictions. We demonstrate that it is possible to identify 'number' neurons, 'string' neurons, and higher level 'text' neurons which are responsible for specific code properties. This could potentially be used to modify neurons responsible for predictions based on incorrect signals. Additionally, the distribution and concentration of the important neurons within different source code embeddings can be used as measures of task complexity, to compare source-code embeddings and guide training choices for transfer learning over similar tasks.

翻译：神经代码智能模型对程序员而言始终是“黑箱”。这种不透明性限制了它们在代码智能任务中的应用，特别是在漏洞检测等场景中，模型对虚假相关性的依赖可能危及安全性。我们提出一种面向神经代码智能模型可解释性的神经元级方法，通过消除网络中高度相似或无关的冗余神经元来简化模型。我们利用探针分类器评估保留的重要神经元——这类分类器常被用于判定神经智能模型潜在表征中是否编码了特定属性。然而，由于代码数据集中标记的重复性和确定性，探针分类的准确率可能被人为放大。因此，我们借鉴自然语言处理中原有的选择性度量方法，通过适配探针记忆效应来构建源代码探针任务。通过神经元分析，我们发现超过95%的神经元与代码智能任务无关，可在不显著损失准确率的情况下被消除。我们进一步将单个及一组重要神经元追溯至可影响模型预测的特定代码属性，并证明能够识别出负责数字属性、字符串属性及高阶文本属性的神经元。这可用于修正基于错误信号的预测神经元。此外，重要神经元在不同源代码嵌入中的分布密度可作为任务复杂度的度量指标，用于比较代码嵌入并指导相似任务迁移学习的训练策略选择。