The characterization of the functions spaces explored by neural networks (NNs) is an important aspect of deep learning theory. In this work, we view a multi-layer NN with arbitrary width as defining a particular hierarchy of reproducing kernel Hilbert spaces (RKHSs), named a Neural Hilbert Ladder (NHL). This allows us to define a function space and a complexity measure that generalize prior results for shallow NNs, and we then examine their theoretical properties and implications in several aspects. First, we prove a correspondence between functions expressed by L-layer NNs and those belonging to L-level NHLs. Second, we prove generalization guarantees for learning an NHL with the complexity measure controlled. Third, corresponding to the training of multi-layer NNs in the infinite-width mean-field limit, we derive an evolution of the NHL characterized as the dynamics of multiple random fields. Fourth, we show examples of depth separation in NHLs under ReLU and quadratic activation functions. Finally, we complement the theory with numerical results to illustrate the learning of RKHS in NN training.

翻译：神经网络（NNs）所探索的函数空间的刻画是深度学习理论的一个重要方面。本文中，我们将任意宽度的多层神经网络视为定义了一类特定的再生核希尔伯特空间（RKHSs）层次结构，称为神经希尔伯特阶梯（NHL）。这使我们能够定义一个函数空间和一个复杂度度量，这些泛化了浅层神经网络的先前结果，随后我们从多个方面考察它们的理论性质及意义。首先，我们证明了由L层NNs表示的函数与属于L级NHLs的函数之间的对应关系。其次，我们证明了在复杂度度量受控条件下学习NHL的泛化保证。第三，对应于无限宽度均场极限下多层NNs的训练，我们推导了NHL的演化过程，其特征在于多个随机场的动力学。第四，我们展示了在ReLU和二次激活函数下NHLs中深度分离的例子。最后，我们通过数值结果对理论进行补充，以说明NN训练中RKHS的学习。