This survey provides an in-depth and explanatory review of the approximation properties of deep neural networks, with a focus on feed-forward and residual architectures. The primary objective is to examine how effectively neural networks approximate target functions and to identify conditions under which they outperform traditional approximation methods. Key topics include the nonlinear, compositional structure of deep networks and the formalization of neural network tasks as optimization problems in regression and classification settings. The survey also addresses the training process, emphasizing the role of stochastic gradient descent and backpropagation in solving these optimization problems, and highlights practical considerations such as activation functions, overfitting, and regularization techniques. Additionally, the survey explores the density of neural networks in the space of continuous functions, comparing the approximation capabilities of deep ReLU networks with those of other approximation methods. It discusses recent theoretical advancements in understanding the expressiveness and limitations of these networks. A detailed error-complexity analysis is also presented, focusing on error rates and computational complexity for neural networks with ReLU and Fourier-type activation functions in the context of bounded target functions with minimal regularity assumptions. Alongside recent known results, the survey introduces new findings, offering a valuable resource for understanding the theoretical foundations of neural network approximation. Concluding remarks and further reading suggestions are provided.

翻译：本综述对深度神经网络的逼近特性进行了深入且解释性的回顾，重点关注前馈和残差架构。主要目标是研究神经网络如何有效地逼近目标函数，并确定其在何种条件下优于传统逼近方法。关键主题包括深度网络的非线性组合结构，以及将神经网络任务形式化为回归和分类场景中的优化问题。本综述还讨论了训练过程，强调随机梯度下降和反向传播在解决这些优化问题中的作用，并突出了实际考虑因素，如激活函数、过拟合和正则化技术。此外，本综述探讨了神经网络在连续函数空间中的稠密性，比较了深度ReLU网络与其他逼近方法的逼近能力。它讨论了在理解这些网络的表达能力和局限性方面的最新理论进展。同时，本文还进行了详细的误差-复杂度分析，重点研究了在具有最小正则性假设的有界目标函数背景下，采用ReLU和傅里叶型激活函数的神经网络的误差率和计算复杂度。除了已知的最新结果外，本综述还介绍了新的发现，为理解神经网络逼近的理论基础提供了宝贵资源。最后提供了结论性评述和进一步阅读建议。