Fully connected deep neural networks are successfully applied to classification and function approximation problems. By minimizing the cost function, i.e., finding the proper weights and biases, models can be built for accurate predictions. The ideal optimization process can achieve global optima. However, do global optima always perform well? If not, how bad can it be? In this work, we aim to: 1) extend the expressive power of shallow neural networks to networks of any depth using a simple trick, 2) construct extremely overfitting deep neural networks that, despite having global optima, still fail to perform well on classification and function approximation problems. Different types of activation functions are considered, including ReLU, Parametric ReLU, and Sigmoid functions. Extensive theoretical analysis has been conducted, ranging from one-dimensional models to models of any dimensionality. Numerical results illustrate our theoretical findings.

翻译：全连接深度神经网络已成功应用于分类与函数逼近问题。通过最小化代价函数（即寻找合适的权重与偏置），可构建实现精确预测的模型。理想的优化过程能够达到全局最优解。然而，全局最优解是否总能表现良好？若否，其性能可能恶化至何种程度？本研究旨在：1）通过简单技巧将浅层神经网络的表达能力扩展至任意深度网络；2）构建极端过拟合的深度神经网络，此类网络虽获得全局最优解，却在分类与函数逼近问题上表现不佳。研究涵盖了多种激活函数类型，包括ReLU、参数化ReLU及Sigmoid函数。我们开展了从一维模型到任意维度模型的理论分析，并通过数值结果验证了理论发现。