In recent years, the neural tangent kernel (NTK) and neural network Gaussian process kernel (NNGP) have given theoreticians tractable limiting cases of fully connected neural networks. However, the property of these kernels are poorly understood for activation functions other than powers of the ReLU. Our main contribution is a characterization of the RKHS of these kernels for activation functions whose only non-smoothness is at zero. This extends existing theory to numerous commonly used activation functions such as SELU, ELU, or LeakyReLU. Additionally, we analyze a broad set of special cases such as missing biases, two-layer networks, or polynomial activations. Our results show that a broad class of not infinitely smooth activations generate equivalent RKHSs at different network depths, depending only on the degree of the non-smoothness up to equivalence. On the other hand, the RKHS generated by polynomial activations depends on the network depth. Finally, we derive results for the smoothness of NNGP sample paths, characterizing the smoothness of infinitely wide neural networks at initialization.

翻译：近年来，神经正切核（NTK）与神经网络高斯过程核（NNGP）为理论研究者提供了全连接神经网络的易处理极限情形。然而，对于除ReLU幂函数之外的激活函数，这些核的性质仍鲜有深入理解。我们的主要贡献在于，针对非光滑点仅在零点的激活函数，刻画了这些核的再生核希尔伯特空间（RKHS）。该理论扩展至众多常用激活函数，例如SELU、ELU或LeakyReLU。此外，我们分析了若干特例，包括缺失偏置、双层网络或多项式激活。结果表明，一类非无限光滑的激活函数在不同网络深度下可生成等价的RKHS，其等价性仅依赖于非光滑性的阶数。另一方面，由多项式激活生成的RKHS则依赖于网络深度。最后，我们推导了NNGP样本路径的光滑性结论，刻画了初始化时无限宽神经网络的光滑程度。