We propose theoretical analyses of a modified natural gradient descent method in the neural network function space based on the eigendecompositions of neural tangent kernel and Fisher information matrix. We firstly present analytical expression for the function learned by this modified natural gradient under the assumptions of Gaussian distribution and infinite width limit. Thus, we explicitly derive the generalization error of the learned neural network function using theoretical methods from eigendecomposition and statistics theory. By decomposing of the total generalization error attributed to different eigenspace of the kernel in function space, we propose a criterion for balancing the errors stemming from training set and the distribution discrepancy between the training set and the true data. Through this approach, we establish that modifying the training direction of the neural network in function space leads to a reduction in the total generalization error. Furthermore, We demonstrate that this theoretical framework is capable to explain many existing results of generalization enhancing methods. These theoretical results are also illustrated by numerical examples on synthetic data.

翻译：我们基于神经正切核和Fisher信息矩阵的特征分解，提出对神经网络函数空间中修正自然梯度下降法的理论分析。首先，在高斯分布和无限宽度假设下，给出了该修正自然梯度法所学函数的解析表达式。进而，利用特征分解和统计学的理论方法，显式推导出所学神经网络函数的泛化误差。通过将总泛化误差分解为核函数函数空间中不同特征空间的贡献，我们提出一个准则来平衡来自训练集的误差和训练集与真实数据之间分布差异带来的误差。通过这种方法，我们证明在函数空间中修改神经网络的训练方向可以降低总泛化误差。此外，我们展示该理论框架能够解释许多现有增强泛化方法的结果。这些理论结果也在合成数据的数值示例中得到验证。