Deep neural networks (DNNs) suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN composed with the SNNs trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs.

翻译：深度神经网络（DNNs）存在频谱偏差问题，即DNNs通常倾向于优先学习函数的低频成分，难以捕捉其高频特征。本文旨在解决这一问题。注意到仅包含低频分量的函数可由浅层神经网络（SNN）——即仅包含少数几层的网络——良好表示。通过观察到低频函数的组合可以有效逼近高频函数，我们提出通过组合多个SNN来学习包含高频分量的函数，其中每个SNN从给定数据中学习特定的低频信息。我们利用多层级深度学习（MGDL）模型实现所提出的思路，该模型是近期提出的一种逐层级增量训练DNN的方法，每一级仅从上一级的残差中学习一个SNN，并与前几级已训练的SNN组合作为特征。我们将MGDL应用于合成数据、流形数据、彩色图像及MNIST数据集，这些数据均以存在高频特征为特点。研究表明，MGDL在表示包含高频信息的函数方面表现卓越。具体而言，每一级学习到的神经网络能有效捕获部分低频信息，使其与前几级学习的SNN组合后能有效表示高频特征。实验结果证实了MGDL在解决DNN固有频谱偏差方面的有效性。通过利用MGDL，我们为克服DNN的频谱偏差限制提供了见解，从而提升了深度学习模型在需要表示高频信息任务中的性能与适用性。本研究证实所提方法为解决DNN频谱偏差问题提供了一个有前景的解决方案。