This article provides a comprehensive understanding of optimization in deep learning, with a primary focus on the challenges of gradient vanishing and gradient exploding, which normally lead to diminished model representational ability and training instability, respectively. We analyze these two challenges through several strategic measures, including the improvement of gradient flow and the imposition of constraints on a network's Lipschitz constant. To help understand the current optimization methodologies, we categorize them into two classes: explicit optimization and implicit optimization. Explicit optimization methods involve direct manipulation of optimizer parameters, including weight, gradient, learning rate, and weight decay. Implicit optimization methods, by contrast, focus on improving the overall landscape of a network by enhancing its modules, such as residual shortcuts, normalization methods, attention mechanisms, and activations. In this article, we provide an in-depth analysis of these two optimization classes and undertake a thorough examination of the Jacobian matrices and the Lipschitz constants of many widely used deep learning modules, highlighting existing issues as well as potential improvements. Moreover, we also conduct a series of analytical experiments to substantiate our theoretical discussions. This article does not aim to propose a new optimizer or network. Rather, our intention is to present a comprehensive understanding of optimization in deep learning. We hope that this article will assist readers in gaining a deeper insight in this field and encourages the development of more robust, efficient, and high-performing models.

翻译：本文全面阐述了深度学习中的优化问题，重点关注梯度消失与梯度爆炸两大挑战，这些问题通常分别导致模型表征能力下降和训练不稳定。我们通过若干策略性措施对这两类挑战进行分析，包括改进梯度流动以及对网络Lipschitz常数施加约束。为便于理解当前优化方法，我们将其分为两类：显式优化与隐式优化。显式优化方法涉及直接调整优化器参数，包括权重、梯度、学习率和权重衰减；隐式优化方法则通过增强网络模块（如残差捷径、归一化方法、注意力机制和激活函数）来改善网络的整体势能地貌。本文对这两类优化进行了深入分析，并系统考察了众多常用深度学习模块的Jacobian矩阵与Lipschitz常数，揭示了现有问题及潜在改进方向。此外，我们还开展了一系列分析性实验以佐证理论讨论。本文并非旨在提出新的优化器或网络结构，而是希望提供对深度学习优化的全面理解。我们期待本文能帮助读者深化对该领域的认知，并推动更鲁棒、高效且高性能模型的发展。