A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at https://github.com/berndprach/NActivation.

翻译：实现安全、可信且可解释的深度学习系统的关键属性在于其鲁棒性：系统输入的微小变化不应导致输出的巨大变化。从数学角度看，这要求网络具有较小的Lipschitz常数。近期多项工作聚焦于如何构建此类Lipschitz网络，通常通过对权重矩阵施加约束来实现。本研究探讨了一个正交方向，即激活函数的作用。研究表明，常用激活函数（如MaxMin）以及所有两段式分段线性激活函数，即使在一维最简单场景下，也会不必要地限制可表示函数的类别。我们进一步引入新型N-激活函数，该函数在理论上比当前流行的激活函数具有更强的表达能力。相关代码已发布至https://github.com/berndprach/NActivation。