Spectral bias is an important observation of neural network training, stating that the network will learn a low frequency representation of the target function before converging to higher frequency components. This property is interesting due to its link to good generalization in over-parameterized networks. However, in low dimensional settings, a severe spectral bias occurs that obstructs convergence to high frequency components entirely. In order to overcome this limitation, one can encode the inputs using a high frequency sinusoidal encoding. Previous works attempted to explain this phenomenon using Neural Tangent Kernel (NTK) and Fourier analysis. However, NTK does not capture real network dynamics, and Fourier analysis only offers a global perspective on the network properties that induce this bias. In this paper, we provide a novel approach towards understanding spectral bias by directly studying ReLU MLP training dynamics. Specifically, we focus on the connection between the computations of ReLU networks (activation regions), and the speed of gradient descent convergence. We study these dynamics in relation to the spatial information of the signal to understand how they influence spectral bias. We then use this formulation to study the severity of spectral bias in low dimensional settings, and how positional encoding overcomes this.

翻译：谱偏置是神经网络训练中的一个重要发现，它表明网络在收敛到高频分量之前，会先学习目标函数的低频表示。这一性质因其与过参数化网络的良好泛化能力相关联而备受关注。然而，在低维场景中，严重的谱偏置会完全阻碍网络向高频分量的收敛。为克服这一局限，可采用高频正弦编码对输入进行编码。现有研究尝试利用神经正切核（NTK）与傅里叶分析解释该现象，但NTK无法捕捉真实网络动力学，而傅里叶分析仅能从全局视角揭示引发偏置的网络特性。本文提出一种全新方法，通过直接研究ReLU MLP的训练动力学来理解谱偏置。具体而言，我们聚焦于ReLU网络的计算过程（激活区域）与梯度下降收敛速度之间的关联，并基于信号空间信息分析这些动力学特性如何影响谱偏置。进而利用该理论框架，探究低维场景中谱偏置的严重程度以及位置编码对此问题的缓解机制。