The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training, via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any non-polynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions which is of independent interest.

翻译：神经正切核（Neural Tangent Kernel，NTK）已成为宽神经网络研究中的一个基本概念。特别地，已知NTK的正定性直接关系到充分宽网络的记忆容量，即通过梯度下降实现训练零损失的可能性。本文将改进先前的研究，并获得关于任意深度前馈网络NTK正定性的一个精确结果。更确切地说，我们将证明：对于任何非多项式激活函数，NTK是严格正定的。我们的结果基于多项式函数的一个新颖刻画，该刻画本身具有独立的意义。