There remains a list of unanswered research questions on deep learning (DL), including the remarkable generalization power of overparametrized neural networks, the efficient optimization performance despite the non-convexity, and the mechanisms behind flat minima in generalization. In this paper, we adopt an information-theoretic perspective to explore the theoretical foundations of supervised classification using deep neural networks (DNNs). Our analysis introduces the concepts of fitting error and model risk, which, together with generalization error, constitute an upper bound on the expected risk. We demonstrate that the generalization errors are bounded by the complexity, influenced by both the smoothness of distribution and the sample size. Consequently, task complexity serves as a reliable indicator of the dataset's quality, guiding the setting of regularization hyperparameters. Furthermore, the derived upper bound fitting error links the back-propagated gradient, Neural Tangent Kernel (NTK), and the model's parameter count with the fitting error. Utilizing the triangle inequality, we establish an upper bound on the expected risk. This bound offers valuable insights into the effects of overparameterization, non-convex optimization, and the flat minima in DNNs.Finally, empirical verification confirms a significant positive correlation between the derived theoretical bounds and the practical expected risk, confirming the practical relevance of the theoretical findings.

翻译：深度学习领域仍存在一系列未解决的研究问题，包括过参数化神经网络卓越的泛化能力、非凸性下高效的优化性能，以及泛化过程中平缓极小值背后的机制。本文采用信息论视角，探讨基于深度神经网络（DNN）的监督分类理论基础。分析中引入拟合误差与模型风险的概念，二者与泛化误差共同构成期望风险的上界。我们证明泛化误差受复杂度约束，且该复杂度同时受分布平滑度与样本量影响。因此，任务复杂度可作为数据集质量的可靠指标，指导正则化超参数的设定。此外，导出的拟合误差上界将反向传播梯度、神经正切核（NTK）及模型参数数量与拟合误差相关联。利用三角不等式，我们建立了期望风险的上界。该界为过参数化、非凸优化及DNN中的平缓极小值提供了重要见解。最后，实证验证表明，导出的理论界与实际期望风险之间存在显著正相关，证实了理论结果的实践相关性。