Randomized neural networks (randomized NNs), where only the terminal layer's weights are optimized constitute a powerful model class to reduce computational time in training the neural network model. At the same time, these models generalize surprisingly well in various regression and classification tasks. In this paper, we give an exact macroscopic characterization (i.e., a characterization in function space) of the generalization behavior of randomized, shallow NNs with ReLU activation (RSNs). We show that RSNs correspond to a generalized additive model (GAM)-typed regression in which infinitely many directions are considered: the infinite generalized additive model (IGAM). The IGAM is formalized as solution to an optimization problem in function space for a specific regularization functional and a fairly general loss. This work is an extension to multivariate NNs of prior work, where we showed how wide RSNs with ReLU activation behave like spline regression under certain conditions and if the input is one-dimensional.

翻译：随机化神经网络（随机化NN）仅优化终端层权重，是一类能有效降低神经网络模型训练计算时间的强大模型。同时，这类模型在各种回归与分类任务中展现出令人惊讶的泛化能力。本文针对采用ReLU激活函数的随机化浅层神经网络（RSN），给出了其泛化行为的精确宏观表征（即函数空间中的表征）。我们证明RSN对应于一种考虑无穷多个方向的广义加性模型（GAM）型回归：即无穷广义加性模型（IGAM）。该IGAM被形式化为函数空间中针对特定正则化泛函及相当一般损失函数的优化问题解。本工作是先前研究向多元神经网络的扩展——先前工作揭示了在特定条件下且输入为一维时，宽随机化ReLU神经网络如何像样条回归般运作。