This article provides a comprehensive understanding of optimization in deep learning, with a primary focus on the challenges of gradient vanishing and gradient exploding, which normally lead to diminished model representational ability and training instability, respectively. We analyze these two challenges through several strategic measures, including the improvement of gradient flow and the imposition of constraints on a network's Lipschitz constant. To help understand the current optimization methodologies, we categorize them into two classes: explicit optimization and implicit optimization. Explicit optimization methods involve direct manipulation of optimizer parameters, including weight, gradient, learning rate, and weight decay. Implicit optimization methods, by contrast, focus on improving the overall landscape of a network by enhancing its modules, such as residual shortcuts, normalization methods, attention mechanisms, and activations. In this article, we provide an in-depth analysis of these two optimization classes and undertake a thorough examination of the Jacobian matrices and the Lipschitz constants of many widely used deep learning modules, highlighting existing issues as well as potential improvements. Moreover, we also conduct a series of analytical experiments to substantiate our theoretical discussions. This article does not aim to propose a new optimizer or network. Rather, our intention is to present a comprehensive understanding of optimization in deep learning. We hope that this article will assist readers in gaining a deeper insight in this field and encourages the development of more robust, efficient, and high-performing models.

翻译：本文旨在全面理解深度学习中的优化问题，重点关注梯度消失与梯度爆炸这两大挑战——前者通常导致模型表征能力下降，后者则引发训练不稳定。我们通过梯度流改进与网络利普希茨常数约束等策略手段对这两个挑战展开分析。为便于理解当前优化方法体系，我们将其分为显式优化与隐式优化两类：显式优化方法涉及对优化器参数的直接调控，包括权重、梯度、学习率与权重衰减；隐式优化方法则着重通过增强网络模块（如残差捷径、归一化方法、注意力机制与激活函数）来改善其整体损失景观。本文深入剖析了这两类优化方法，系统考察了多种常用深度学习模块的雅可比矩阵与利普希茨常数，揭示了现有问题与改进潜力。此外，我们通过系列分析实验验证了理论讨论。本文并非旨在提出新型优化器或网络结构，而是期望呈现对深度学习优化的系统性理解。我们相信，本文能够帮助读者深化对该领域的认知，并推动更稳健、高效且高性能模型的发展。