In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or scaling, results in a corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element-wise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.

翻译：在许多信息处理系统中，确保输入的任何变化（无论是平移还是缩放）都会导致系统响应产生相应的变化，这通常是期望的特性。尽管深度神经网络正逐步取代所有传统的自动处理方法，但它们令人意外地无法保证这种归一化等变性（缩放+平移）属性，这在许多应用中可能是有害的。为解决这一问题，我们提出了一种方法，用于调整现有神经网络，使其在设计上就具备归一化等变性。我们的主要观点是：不仅普通的卷积层，而且包括逐元素应用于预激活神经元的ReLU（修正线性单元）在内的所有激活函数，都应从神经网络中完全移除，并替换为条件更优的替代方案。为此，我们引入了仿射约束卷积和通道级排序池化层作为替代，并证明这两种架构修改确实能保持归一化等变性且性能无损失。图像去噪的实验结果表明，归一化等变神经网络除了具有更好的条件性外，还能在噪声水平间提供更好的泛化能力。