This paper presents two models of neural-networks and their training applicable to neural networks of arbitrary width, depth and topology, assuming only finite-energy neural activations; and a novel representor theory for neural networks in terms of a matrix-valued kernel. The first model is exact (un-approximated) and global, casting the neural network as an elements in a reproducing kernel Banach space (RKBS); we use this model to provide tight bounds on Rademacher complexity. The second model is exact and local, casting the change in neural network function resulting from a bounded change in weights and biases (ie. a training step) in reproducing kernel Hilbert space (RKHS) in terms of a local-intrinsic neural kernel (LiNK). This local model provides insight into model adaptation through tight bounds on Rademacher complexity of network adaptation. We also prove that the neural tangent kernel (NTK) is a first-order approximation of the LiNK kernel. Finally, and noting that the LiNK does not provide a representor theory for technical reasons, we present an exact novel representor theory for layer-wise neural network training with unregularized gradient descent in terms of a local-extrinsic neural kernel (LeNK). This representor theory gives insight into the role of higher-order statistics in neural network training and the effect of kernel evolution in neural-network kernel models. Throughout the paper (a) feedforward ReLU networks and (b) residual networks (ResNet) are used as illustrative examples.

翻译：本文提出了两种适用于任意宽度、深度和拓扑结构的神经网络及其训练模型（仅假设神经元激活具有有限能量），以及一种基于矩阵值核的神经网络新型表示理论。第一个模型是精确（未经近似）且全局性的，将神经网络视为再生核巴拿赫空间（RKBS）中的元素；我们利用该模型给出拉德马赫复杂度的紧致界。第二个模型是精确且局部性的，在再生核希尔伯特空间（RKHS）中通过局部内禀神经核（LiNK），将权重与偏置的有界变化（即训练步骤）引起的神经网络函数变化进行建模。该局部模型通过对网络适应的拉德马赫复杂度建立紧致界，为模型适应机制提供了理论洞见。我们同时证明了神经正切核（NTK）是LiNK核的一阶近似。最后，鉴于LiNK因技术限制未能构建表示理论，我们提出了一种基于局部外禀神经核（LeNK）的、针对无正则化梯度下降逐层神经网络训练的精确新型表示理论。该表示理论揭示了高阶统计量在神经网络训练中的作用，以及核演化在神经网络核模型中的影响。全文以（a）前馈ReLU网络与（b）残差网络（ResNet）作为示例进行阐释。