Since the control of the Lipschitz constant has a great impact on the training stability, generalization, and robustness of neural networks, the estimation of this value is nowadays a real scientific challenge. In this paper we introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory and a new alternative to the Power iteration. Called the Gram iteration, our approach exhibits a superlinear convergence. First, we show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. Then, it proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches.

翻译：由于Lipschitz常数的控制对神经网络的训练稳定性、泛化能力和鲁棒性具有重要影响，该值的估计已成为当前真正的科学挑战。本文利用循环矩阵理论并结合一种替代幂迭代的新方法，提出了一种精确、快速且可微的卷积层谱范数上界。我们提出的Gram迭代方法呈现出超线性收敛特性。首先，通过一系列综合性实验表明，本方法在精度、计算成本和可扩展性方面均优于其他现有最优方法。其次，该方法在卷积神经网络的Lipschitz正则化中展现出高效性能，与同期方法相比取得了具有竞争力的结果。