Lipschitz continuity is a simple yet pivotal functional property of any predictive model that lies at the core of its robustness, generalisation, and adversarial vulnerability. Our aim is to thoroughly investigate and characterise the Lipschitz behaviour of the functions learned via neural networks. Despite the significant tightening of the bounds in the recent years, precisely estimating the Lipschitz constant continues to be a practical challenge and tight theoretical analyses, similarly, remain intractable. Therefore, we shift our perspective and instead attempt to uncover insights about the nature of Lipschitz constant of neural networks functions -- by relying on the simplest and most general upper and lower bounds. We carry out an empirical investigation in a range of different settings (architectures, losses, optimisers, label noise, etc.), which reveals several fundamental and intriguing traits of the Lipschitz continuity of neural networks functions, In particular, we identify a remarkable double descent trend in both upper and lower bounds to the Lipschitz constant which tightly aligns with the typical double descent trend in the test loss.

翻译：Lipschitz连续性是一个简单但关键的预测模型函数性质，它位于模型鲁棒性、泛化能力和对抗脆弱性的核心。本文旨在深入研究和刻画神经网络所学函数的Lipschitz行为。尽管近年来上下界得到了显著收紧，精确估计Lipschitz常数仍是一个实际挑战，同样，严格的理论分析也仍然难以处理。因此，我们转变视角，转而依赖最简单且最通用的上下界，尝试揭示神经网络函数Lipschitz常数的本质特征。我们在多种不同设置（架构、损失函数、优化器、标签噪声等）下开展实证研究，揭示了神经网络函数Lipschitz连续性的若干基本且有趣的特性。特别地，我们发现在Lipschitz常数的上下界中均存在显著的"双下降"趋势，该趋势与测试损失中典型的"双下降"趋势高度吻合。