Since the control of the Lipschitz constant has a great impact on the training stability, generalization, and robustness of neural networks, the estimation of this value is nowadays a real scientific challenge. In this paper we introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory and a new alternative to the Power iteration. Called the Gram iteration, our approach exhibits a superlinear convergence. First, we show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. Then, it proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches. Code is available at https://github.com/blaisedelattre/lip4conv.

翻译：由于Lipschitz常数的控制对神经网络的训练稳定性、泛化能力和鲁棒性具有重要影响，该值的估计已成为当前科学研究的一大挑战。本文利用循环矩阵理论和一种新的幂迭代替代方法，提出了一种精确、快速且可微的卷积层谱范数上界。该方法命名为"格拉姆迭代"，具有超线性收敛特性。首先，通过一系列综合实验表明，该方法在精度、计算成本和可扩展性方面均优于当前最先进方法；其次，该方法在卷积神经网络的Lipschitz正则化中展现出高效性，其效果与同期方法相比具有竞争力。代码见https://github.com/blaisedelattre/lip4conv。