In this work, we propose a novel activation mechanism aimed at establishing layer-level activation (LayerAct) functions for CNNs with BatchNorm. These functions are designed to be more noise-robust compared to existing element-level activation functions by reducing the layer-level fluctuation of the activation outputs due to shift in inputs. Moreover, the LayerAct functions achieve this noise-robustness independent of the activation's saturation state, which limits the activation output space and complicates efficient training. We present an analysis and experiments demonstrating that LayerAct functions exhibit superior noise-robustness compared to element-level activation functions, and empirically show that these functions have a zero-like mean activation. Experimental results with three clean and three out-of-distribution benchmark datasets for image classification tasks show that LayerAct functions excel in handling noisy datasets, outperforming element-level activation functions, while the performance on clean datasets is also superior in most cases.

翻译：本文提出了一种新颖的激活机制，旨在为带有BatchNorm的CNN建立层级别激活（LayerAct）函数。与现有元素级激活函数相比，这些函数通过减少因输入偏移导致的激活输出层级别波动，具有更强的噪声鲁棒性。此外，LayerAct函数在实现这种噪声鲁棒性时不受激活饱和状态的影响，而激活饱和状态会限制激活输出空间并使高效训练复杂化。我们通过分析和实验证明，LayerAct函数相比元素级激活函数表现出更优的噪声鲁棒性，并实验表明这些函数具有近似零均值的激活特性。针对图像分类任务，在三个干净数据集和三个分布外基准数据集上的实验结果显示，LayerAct函数在处理噪声数据集方面表现优异，优于元素级激活函数；同时，在干净数据集上，其性能在多数情况下也更为优越。