Lipschitz-constrained neural networks have several advantages over unconstrained ones and can be applied to a variety of problems, making them a topic of attention in the deep learning community. Unfortunately, it has been shown both theoretically and empirically that they perform poorly when equipped with ReLU activation functions. By contrast, neural networks with learnable 1-Lipschitz linear splines are known to be more expressive. In this paper, we show that such networks correspond to global optima of a constrained functional optimization problem that consists of the training of a neural network composed of 1-Lipschitz linear layers and 1-Lipschitz freeform activation functions with second-order total-variation regularization. Further, we propose an efficient method to train these neural networks. Our numerical experiments show that our trained networks compare favorably with existing 1-Lipschitz neural architectures.

翻译：利普希茨约束神经网络相比无约束网络具有多项优势，可应用于多种问题，因而成为深度学习领域的研究热点。然而，理论和实践经验均表明，当采用ReLU激活函数时，此类网络性能欠佳。相比之下，配备可学习1-利普希茨线性样条的神经网络具有更强的表达能力。本文证明，这类网络对应约束泛函优化问题的全局最优解——该优化问题涉及由1-利普希茨线性层和具有二阶全变分正则化的1-利普希茨自由形式激活函数组成的神经网络训练过程。我们进一步提出了一种高效训练此类神经网络的方法。数值实验表明，我们训练的网络在性能上优于现有1-利普希茨神经架构。