Neural tangent kernels (NTKs) are a powerful tool for analyzing deep, non-linear neural networks. In the infinite-width limit, NTKs can easily be computed for most common architectures, yielding full analytic control over the training dynamics. However, at infinite width, important properties of training such as NTK evolution or feature learning are absent. Nevertheless, finite width effects can be included by computing corrections to the Gaussian statistics at infinite width. We introduce Feynman diagrams for computing finite-width corrections to NTK statistics. These dramatically simplify the necessary algebraic manipulations and enable the computation of layer-wise recursion relations for arbitrary statistics involving preactivations, NTKs and certain higher-derivative tensors (dNTK and ddNTK) required to predict the training dynamics at leading order. We demonstrate the feasibility of our framework by extending stability results for deep networks from preactivations to NTKs and proving the absence of finite-width corrections for scale-invariant nonlinearities such as ReLU on the diagonal of the Gram matrix of the NTK. We numerically implement the complete set of equations necessary to compute the first-order corrections for arbitrary inputs and demonstrate that the results follow the statistics of sampled neural networks for widths $n\gtrsim 20$.

翻译：神经正切核（NTK）是分析深度非线性神经网络的强大工具。在无限宽度极限下，对于大多数常见架构可以轻松计算NTK，从而实现对训练动态的完全解析控制。然而，在无限宽度条件下，训练的重要特性（如NTK演化或特征学习）并不存在。尽管如此，可以通过计算无限宽度高斯统计量的修正来纳入有限宽度效应。本文引入用于计算NTK统计量有限宽度修正的费曼图。这些方法极大简化了必要的代数运算，并使得能够计算涉及预激活值、NTK以及预测主导阶训练动态所需的某些高阶导数张量（dNTK和ddNTK）的任意统计量的逐层递推关系。我们通过将深度网络的稳定性结果从预激活值推广至NTK，并证明对于尺度不变非线性函数（如ReLU）在NTK格拉姆矩阵对角线上不存在有限宽度修正，从而验证了本框架的可行性。我们数值实现了计算任意输入一阶修正所需的完整方程组，并证明当宽度$n\gtrsim 20$时，计算结果与采样神经网络的统计特性相符。