Activation functions are essential to deep learning networks. Popular and versatile activation functions are mostly monotonic functions, some non-monotonic activation functions are being explored and show promising performance. But by introducing non-monotonicity, they also alter the positive input, which is proved to be unnecessary by the success of ReLU and its variants. In this paper, we double down on the non-monotonic activation functions' development and propose the Saturated Gaussian Error Linear Units by combining the characteristics of ReLU and non-monotonic activation functions. We present three new activation functions built with our proposed method: SGELU, SSiLU, and SMish, which are composed of the negative portion of GELU, SiLU, and Mish, respectively, and ReLU's positive portion. The results of image classification experiments on CIFAR-100 indicate that our proposed activation functions are highly effective and outperform state-of-the-art baselines across multiple deep learning architectures.

翻译：激活函数对深度学习网络至关重要。目前流行且通用的激活函数多为单调函数，部分非单调激活函数正在被探索并展现出优异的性能。然而，引入非单调性的同时，它们也会改变正输入信号，而ReLU及其变体的成功已经证明这种改变并非必要。本文进一步深化了对非单调激活函数的研究，通过结合ReLU与非单调激活函数的特性，提出了饱和高斯误差线性单元（Saturated Gaussian Error Linear Units）。我们基于所提出的方法构建了三种新型激活函数：SGELU、SSiLU和SMish，它们分别由GELU、SiLU和Mish的负部分与ReLU的正部分组合而成。在CIFAR-100数据集上的图像分类实验结果表明，我们所提出的激活函数具有高效性，且在多种深度学习架构上均优于当前最优基线方法。