Deep learning techniques are increasingly applied to scientific problems, where the precision of networks is crucial. Despite being deemed as universal function approximators, neural networks, in practice, struggle to reduce the prediction errors below $O(10^{-5})$ even with large network size and extended training iterations. To address this issue, we developed the multi-stage neural networks that divides the training process into different stages, with each stage using a new network that is optimized to fit the residue from the previous stage. Across successive stages, the residue magnitudes decreases substantially and follows an inverse power-law relationship with the residue frequencies. The multi-stage neural networks effectively mitigate the spectral biases associated with regular neural networks, enabling them to capture the high frequency feature of target functions. We demonstrate that the prediction error from the multi-stage training for both regression problems and physics-informed neural networks can nearly reach the machine-precision $O(10^{-16})$ of double-floating point within a finite number of iterations. Such levels of accuracy are rarely attainable using single neural networks alone.

翻译：深度学习技术日益应用于科学问题，其中网络的精度至关重要。尽管神经网络被视作通用函数逼近器，但在实践中，即使采用大规模网络和大量训练迭代，其预测误差也难以降至$O(10^{-5})$以下。为解决这一问题，我们开发了多阶段神经网络，将训练过程分为不同阶段，每个阶段使用一个新网络来优化拟合前一阶段的残差。在连续阶段中，残差幅值显著降低，且与残差频率服从逆幂律关系。多阶段神经网络有效缓解了常规神经网络相关的谱偏差，使其能够捕捉目标函数的高频特征。我们证明，对于回归问题和物理信息神经网络，多阶段训练的预测误差在有限迭代次数内几乎可达到双精度浮点数的机器精度$O(10^{-16})$。单神经网络本身几乎无法实现如此高的精度水平。