As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance on the training data and unseen data). Focusing on the stochastic nature of training, we first present a theoretical analysis in which the bound of generalization gap depends on what we call inconsistency and instability of model outputs, which can be estimated on unlabeled data. Our empirical study based on this analysis shows that instability and inconsistency are strongly predictive of generalization gap in various settings. In particular, our finding indicates that inconsistency is a more reliable indicator of generalization gap than the sharpness of the loss landscape. Furthermore, we show that algorithmic reduction of inconsistency leads to superior performance. The results also provide a theoretical basis for existing methods such as co-distillation and ensemble.

翻译：由于深度神经网络具有高度表达能力，找到泛化差距（训练数据与未见数据性能之差）较小的解至关重要。聚焦于训练的随机性，我们首先提出一项理论分析，其中泛化差距的界限取决于我们称之为模型输出的不一致性和不稳定性，这些指标可在无标注数据上估计。基于此分析的实证研究表明，在不同设置下，不稳定性和不一致性能强有力地预测泛化差距。特别地，我们的发现指出，与损失景观的尖锐度相比，不一致性是泛化差距更可靠的指标。此外，我们证明算法性地降低不一致性可带来卓越性能。这些结果也为协同蒸馏和集成等现有方法提供了理论基础。