Many popular feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model's output with respect to its inputs. While these methods can indicate which input features may be important for the model's prediction, they reveal little about the inner workings of the model itself. In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. We implement this generalized algorithm, evaluate it on synthetic datasets to better understand the statistics it computes, and apply it to study BERT's behavior on the subject-verb number agreement task (SVA). With this method, we (a) validate that the amount of gradient flow through a component of a model reflects its importance to a prediction and (b) for SVA, identify which pathways of the self-attention mechanism are most important.

翻译：许多用于解释深度神经网络的流行特征归因方法依赖计算模型输出相对于输入的梯度。尽管这些方法能指示哪些输入特征对模型预测可能重要，但它们对模型内部运作方式的揭示甚少。本文观察到，模型的梯度计算是一种更通用的基于半环（semirings）公式的特例。这一观察使我们能够泛化反向传播算法，高效计算神经网络梯度图的其他可解释性统计量，例如最高权重路径和熵。我们实现了该泛化算法，在合成数据集上评估以更好地理解其计算的统计量，并将其应用于研究BERT在主语-动词数一致任务（SVA）中的行为。通过此方法，我们（a）验证了通过模型某个组件的梯度流量反映了该组件对预测的重要性，以及（b）针对SVA识别了自注意力机制中最关键的路径。