Since the control of the Lipschitz constant has a great impact on the training stability, generalization, and robustness of neural networks, the estimation of this value is nowadays a real scientific challenge. In this paper we introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory and a new alternative to the Power iteration. Called the Gram iteration, our approach exhibits a superlinear convergence. First, we show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. Then, it proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches. Code is available at https://github.com/blaisedelattre/lip4conv.

翻译：由于Lipschitz常数的控制对神经网络的训练稳定性、泛化能力和鲁棒性具有重要影响，该值的估计已成为当前科学研究的重大挑战。本文利用循环矩阵理论及幂迭代的新替代方法，提出了针对卷积层谱范数的精确、快速且可微分的上界。该方法被称为Gram迭代，表现出超线性收敛特性。首先，通过一系列综合性实验表明，本方法在精度、计算成本和可扩展性方面均优于现有最优方法。其次，该方法在卷积神经网络的Lipschitz正则化中展现出高效性，与同期方法相比具有竞争力。代码已开源在https://github.com/blaisedelattre/lip4conv。