In this paper, we identify and analyze a recurring training loss pattern, which we term the \textit{Epochal Sawtooth Phenomenon (ESP)}, commonly observed during training with adaptive gradient-based optimizers, particularly Adam optimizer. This pattern is characterized by a sharp drop in loss at the beginning of each epoch, followed by a gradual increase, resulting in a sawtooth-shaped loss curve. Through empirical observations, we demonstrate that while this effect is most pronounced with Adam, it persists, although less severely, with other optimizers such as RMSProp. We empirically analyze the mechanisms underlying ESP, focusing on key factors such as Adam's $\beta$ parameters, batch size, data shuffling, and sample replacement. Our analysis shows that ESP arises from adaptive learning rate adjustments controlled by the second moment estimate. Additionally, we identify the ``immediate re-exposure to samples'' effect during data shuffling, which causes the model to learn or memorize more at the beginning of each epoch. We also find that smaller values of $\beta_2$ exacerbate ESP but can act as a form of regularization. While ESP is not necessarily indicative of overfitting, higher model capacity can amplify the phenomenon. To further support our analysis, we replicate ESP through a high-dimensional quadratic minimization task. We demonstrate that ESP can emerge even in simple optimization scenarios, reinforcing the generality of this pattern. The code for reproducing our experiments is available at https://github.com/qiliuchn/training-loss-pattern.

翻译：本文识别并分析了一种反复出现的训练损失模式，我们称之为\textit{时代性锯齿现象（ESP）}，该现象在使用自适应梯度优化器（尤其是Adam优化器）进行训练时普遍存在。该模式的特征是每个训练时代开始时损失急剧下降，随后逐渐上升，形成锯齿状的损失曲线。通过实证观察，我们证明虽然Adam优化器下该效应最为显著，但在其他优化器（如RMSProp）中该现象依然存在，只是程度较轻。我们对ESP的潜在机制进行了实证分析，重点关注Adam的$\beta$参数、批处理大小、数据洗牌和样本替换等关键因素。分析表明，ESP源于由二阶矩估计控制的自适应学习率调整。此外，我们识别出数据洗牌过程中的“样本即时再暴露”效应，该效应导致模型在每个时代开始时学习或记忆更多信息。我们还发现较小的$\beta_2$值会加剧ESP，但可起到某种正则化作用。虽然ESP不一定意味着过拟合，但更高的模型容量会放大该现象。为进⼀步支持我们的分析，我们通过高维二次最小化任务复现了ESP。我们证明即使在简单的优化场景中ESP也可能出现，这强化了该模式的普遍性。实验复现代码可在https://github.com/qiliuchn/training-loss-pattern获取。