We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

翻译：我们从傅里叶分析视角研究了深度神经网络（DNN）的训练过程。在高维基准数据集（如MNIST/CIFAR10）和深层神经网络（如VGG16）中，我们发现了一个极具普遍性的频率原理（F-Principle）——DNN通常按照从低频到高频的顺序拟合目标函数。DNN的这一频率原理与多数传统迭代数值方法（如雅可比迭代法）的行为截然相反——对于各类科学计算问题，后者在处理高频分量时收敛速度更快。通过简洁的理论推导，我们阐明该频率原理源于常用激活函数的正则性。频率原理揭示了DNN存在隐式偏差：其倾向于用低频函数拟合训练数据。这一认知解释了DNN在大多数真实数据集上泛化能力优异，而在parity函数或随机化数据集上泛化能力欠佳的现象。