In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or scaling, results in a corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element-wise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.

翻译：在许多信息处理系统中，确保输入的任何变化（无论是平移还是缩放）都能导致系统响应发生相应变化，这通常是理想特性。尽管深度神经网络正逐渐取代所有传统自动处理方法，但令人惊讶的是，它们并不能保证这种归一化等变性（尺度+平移）属性，而这在许多应用中可能是有害的。为解决这一问题，我们提出了一种方法，通过设计使现有神经网络具备归一化等变性。我们的主要论点是：不仅普通的卷积层，而且所有逐元素应用于预激活神经元的激活函数（包括ReLU，即修正线性单元），都应完全从神经网络中移除，并替换为条件更优的替代方案。为此，我们引入仿射约束卷积和通道级排序池化层作为替代，并证明这两种架构修改在不损失性能的前提下确实保持了归一化等变性。图像去噪的实验结果表明，归一化等变神经网络除了具有更好的条件性外，还能在不同噪声水平下提供更好的泛化能力。