This paper introduces a new parameterization of deep neural networks (both fully-connected and convolutional) with guaranteed Lipschitz bounds, i.e. limited sensitivity to perturbations. The Lipschitz guarantees are equivalent to the tightest-known bounds based on certification via a semidefinite program (SDP), which does not scale to large models. In contrast to the SDP approach, we provide a ``direct'' parameterization, i.e. a smooth mapping from $\mathbb R^N$ onto the set of weights of Lipschitz-bounded networks. This enables training via standard gradient methods, without any computationally intensive projections or barrier terms. The new parameterization can equivalently be thought of as either a new layer type (the \textit{sandwich layer}), or a novel parameterization of standard feedforward networks with parameter sharing between neighbouring layers. Finally, the comprehensive set of experiments on image classification shows that sandwich layers outperform previous approaches on both empirical and certified robust accuracy.

翻译：本文提出了一种全新的深度神经网络（包括全连接网络和卷积网络）参数化方法，可保证其具有Lipschitz界（即对扰动的灵敏度有限）。该Lipschitz保证等价于基于半定规划（SDP）验证方法所得到的最紧已知界，但后者无法扩展至大型模型。与SDP方法不同，我们提供了"直接"参数化方法，即从$\mathbb R^N$到有界Lipschitz网络权重集合的光滑映射。这使得可通过标准梯度方法进行训练，无需任何计算密集的投影或障碍项。该新参数化方法可等价视为一种新型层结构（即"夹层层"），或是相邻层共享参数的经典前馈网络的新型参数化。最后，在图像分类任务上的全面实验表明，夹层层在经验鲁棒精度和经认证的鲁棒精度上均优于现有方法。