This paper studies the properties of solutions to multi-task shallow ReLU neural network learning problems, wherein the network is trained to fit a dataset with minimal sum of squared weights. Remarkably, the solutions learned for each individual task resemble those obtained by solving a kernel method, revealing a novel connection between neural networks and kernel methods. It is known that single-task neural network training problems are equivalent to minimum norm interpolation problem in a non-Hilbertian Banach space, and that the solutions of such problems are generally non-unique. In contrast, we prove that the solutions to univariate-input, multi-task neural network interpolation problems are almost always unique, and coincide with the solution to a minimum-norm interpolation problem in a Sobolev (Reproducing Kernel) Hilbert Space. We also demonstrate a similar phenomenon in the multivariate-input case; specifically, we show that neural network learning problems with large numbers of diverse tasks are approximately equivalent to an $\ell^2$ (Hilbert space) minimization problem over a fixed kernel determined by the optimal neurons.

翻译：本文研究了多任务浅层ReLU神经网络学习问题解的性质，其中网络通过最小化权重平方和来拟合数据集。值得注意的是，针对每个独立任务学习得到的解与通过核方法求解获得的解相似，这揭示了神经网络与核方法之间的新颖联系。已知单任务神经网络训练问题等价于非希尔伯特巴拿赫空间中的最小范数插值问题，且此类问题的解通常不具唯一性。与之相反，我们证明了单变量输入的多任务神经网络插值问题的解几乎总是唯一的，并且与索伯列夫（再生核）希尔伯特空间中最小范数插值问题的解一致。我们还在多变量输入情形中证明了类似现象；具体而言，我们表明具有大量多样化任务的神经网络学习问题近似等价于在由最优神经元确定的固定核上进行的$\ell^2$（希尔伯特空间）最小化问题。