In many research fields in artificial intelligence, it has been shown that deep neural networks are useful to estimate unknown functions on high dimensional input spaces. However, their generalization performance is not yet completely clarified from the theoretical point of view because they are nonidentifiable and singular learning machines. Moreover, a ReLU function is not differentiable, to which algebraic or analytic methods in singular learning theory cannot be applied. In this paper, we study a deep ReLU neural network in overparametrized cases and prove that the Bayesian free energy, which is equal to the minus log marginal likelihoodor the Bayesian stochastic complexity, is bounded even if the number of layers are larger than necessary to estimate an unknown data-generating function. Since the Bayesian generalization error is equal to the increase of the free energy as a function of a sample size, our result also shows that the Bayesian generalization error does not increase even if a deep ReLU neural network is designed to be sufficiently large or in an opeverparametrized state.

翻译：在人工智能的诸多研究领域中，深度神经网络已被证明能有效估计高维输入空间中的未知函数。然而，由于其作为非正则且奇异的学习机器，其泛化性能尚未从理论层面得到彻底阐明。此外，ReLU函数不可微，奇异学习理论中的代数或分析方法无法直接应用。本文研究了过参数化情形下的深度ReLU神经网络，并证明即使网络层数超过估计未知数据生成函数所需的数量，贝叶斯自由能（等同于负对数边际似然或贝叶斯随机复杂度）仍保持有界。由于贝叶斯泛化误差等价于自由能作为样本量函数的增量，我们的结果进一步表明：即使深度ReLU神经网络被设计得过于庞大或处于过参数化状态，其贝叶斯泛化误差也不会增大。