Despite exceptional achievements, training neural networks remains computationally expensive and is often plagued by instabilities that can degrade convergence. While learning rate schedules can help mitigate these issues, finding optimal schedules is time-consuming and resource-intensive. This work explores theoretical issues concerning training stability in the constant-learning-rate (i.e., without schedule) and small-batch-size regime. Surprisingly, we show that the composition order of gradient updates affects stability and convergence in gradient-based optimizers. We illustrate this new line of thinking using backward-SGD, which produces parameter iterates at each step by reverting the usual forward composition order of batch gradients. Our theoretical analysis shows that in contractive regions (e.g., around minima) backward-SGD converges to a point while the standard forward-SGD generally only converges to a distribution. This leads to improved stability and convergence which we demonstrate experimentally. While full backward-SGD is computationally intensive in practice, it highlights that the extra freedom of modifying the usual iteration composition by reusing creatively previous batches at each optimization step may have important beneficial effects in improving training. Our experiments provide a proof of concept supporting this phenomenon. To our knowledge, this represents a new and unexplored avenue in deep learning optimization.

翻译：尽管取得了卓越成就，神经网络训练仍然计算成本高昂，且常受不稳定性困扰，这些不稳定性可能损害收敛效果。虽然学习率调度策略有助于缓解此类问题，但寻找最优调度方案耗时且资源密集。本研究探讨了恒定学习率（即无调度）与小批量训练机制中训练稳定性的理论问题。令人惊讶的是，我们发现梯度更新的组合顺序会影响基于梯度的优化器的稳定性与收敛性。我们通过反向SGD阐释了这一新思路，该方法通过逆转批次梯度的常规前向组合顺序，在每一步生成参数迭代值。理论分析表明，在收缩区域（例如极小值附近）反向SGD会收敛至单点，而标准前向SGD通常仅收敛至分布。这带来了稳定性与收敛性的提升，我们通过实验验证了该结论。虽然完整的反向SGD在实践中计算强度较大，但它揭示出：通过在每次优化步骤中创造性地复用先前批次数据来修改常规迭代组合方式，这种额外的自由度可能对改善训练产生重要积极影响。我们的实验为支持这一现象提供了概念验证。据我们所知，这代表了深度学习优化领域中一条全新且尚未探索的研究路径。