This paper explores the generalization characteristics of iterative learning algorithms with bounded updates for non-convex loss functions, employing information-theoretic techniques. Our key contribution is a novel bound for the generalization error of these algorithms with bounded updates. Our approach introduces two main novelties: 1) we reformulate the mutual information as the uncertainty of updates, providing a new perspective, and 2) instead of using the chaining rule of mutual information, we employ a variance decomposition technique to decompose information across iterations, allowing for a simpler surrogate process. We analyze our generalization bound under various settings and demonstrate improved bounds. To bridge the gap between theory and practice, we also examine the previously observed scaling behavior in large language models. Ultimately, our work takes a further step for developing practical generalization theories.

翻译：本文采用信息论方法，研究了具有有界更新的迭代学习算法在非凸损失函数下的泛化特性。我们的主要贡献是针对具有有界更新的迭代学习算法，提出了一个新的泛化误差界。该方法包含两项核心创新：1) 将互信息重新表述为更新的不确定性，提供了新的视角；2) 采用方差分解技术替代互信息的链式法则，将信息在迭代间进行分解，从而构建更简洁的替代过程。我们在多种设定下分析了该泛化界，并证明了其改进效果。为弥合理论与实践的差距，我们还考察了先前在大语言模型中观测到的标度行为。最终，该工作为发展实用化泛化理论迈出了坚实的一步。