The generalization performance of deep neural networks with regard to the optimization algorithm is one of the major concerns in machine learning. This performance can be affected by various factors. In this paper, we theoretically prove that the Lipschitz constant of a loss function is an important factor to diminish the generalization error of the output model obtained by Adam or AdamW. The results can be used as a guideline for choosing the loss function when the optimization algorithm is Adam or AdamW. In addition, to evaluate the theoretical bound in a practical setting, we choose the human age estimation problem in computer vision. For assessing the generalization better, the training and test datasets are drawn from different distributions. Our experimental evaluation shows that the loss function with lower Lipschitz constant and maximum value improves the generalization of the model trained by Adam or AdamW.

翻译：深度神经网络的泛化性能与优化算法的关系是机器学习领域的主要关注点之一。这一性能可能受到多种因素的影响。本文从理论上证明，损失函数的Lipschitz常数是降低Adam或AdamW优化器所得输出模型泛化误差的重要因素。该结论可作为选择损失函数时的指导准则（当优化算法为Adam或AdamW时）。此外，为在实际场景中评估该理论边界，我们选取计算机视觉中的人类年龄估计问题作为验证。为更好地评估泛化性能，训练集与测试集采样自不同分布。实验评估表明，具有较低Lipschitz常数和最大值的损失函数能够提升Adam或AdamW训练模型的泛化能力。