Neuron Interpretation has gained traction in the field of interpretability, and have provided fine-grained insights into what a model learns and how language knowledge is distributed amongst its different components. However, the lack of evaluation benchmark and metrics have led to siloed progress within these various methods, with very little work comparing them and highlighting their strengths and weaknesses. The reason for this discrepancy is the difficulty of creating ground truth datasets, for example, many neurons within a given model may learn the same phenomena, and hence there may not be one correct answer. Moreover, a learned phenomenon may spread across several neurons that work together -- surfacing these to create a gold standard challenging. In this work, we propose an evaluation framework that measures the compatibility of a neuron analysis method with other methods. We hypothesize that the more compatible a method is with the majority of the methods, the more confident one can be about its performance. We systematically evaluate our proposed framework and present a comparative analysis of a large set of neuron interpretation methods. We make the evaluation framework available to the community. It enables the evaluation of any new method using 20 concepts and across three pre-trained models.The code is released at https://github.com/fdalvi/neuron-comparative-analysis

翻译：神经元解释方法在可解释性领域日益受到关注，为理解模型所学内容以及语言知识在不同组件中的分布提供了细粒度的洞见。然而，由于缺乏评估基准和度量标准，这些方法的发展各自为政，鲜有工作对其进行横向比较并揭示各自的优劣。造成这一差距的原因在于构建真实标注数据集的困难——例如，给定模型中的多个神经元可能学习到相同的现象，因此不存在唯一正确答案。此外，所学现象可能分散在多个协同工作的神经元中，将这种协同关系显式化以构建黄金标准极具挑战性。本文提出一个评估框架，用于衡量神经元分析方法与其他方法的兼容性。我们假设，一种方法与大多数方法兼容性越高，其性能就越可信。我们系统性地评估了所提框架，并对大量神经元解释方法进行了比较分析。我们将该评估框架开源供学界使用，支持基于20个概念和三个预训练模型评估任意新方法。代码发布于：https://github.com/fdalvi/neuron-comparative-analysis