State-of-the-art techniques for enhancing robustness of deep networks mostly rely on empirical risk minimization with suitable data augmentation. In this paper, we propose a complementary approach motivated by communication theory, aimed at enhancing the signal-to-noise ratio at the output of a neural network layer via neural competition during learning and inference. In addition to minimization of a standard end-to-end cost, neurons compete to sparsely represent layer inputs by maximization of a tilted exponential (TEXP) objective function for the layer. TEXP learning can be interpreted as maximum likelihood estimation of matched filters under a Gaussian model for data noise. Inference in a TEXP layer is accomplished by replacing batch norm by a tilted softmax, which can be interpreted as computation of posterior probabilities for the competing signaling hypotheses represented by each neuron. After providing insights via simplified models, we show, by experimentation on standard image datasets, that TEXP learning and inference enhances robustness against noise and other common corruptions, without requiring data augmentation. Further cumulative gains in robustness against this array of distortions can be obtained by appropriately combining TEXP with data augmentation techniques.

翻译：提升深度网络鲁棒性的前沿技术大多依赖于经验风险最小化与适当的数据增强。本文提出一种受通信理论启发的互补方法，旨在通过学习和推理过程中的神经竞争来增强神经网络层输出的信噪比。除最小化标准端到端损失外，神经元通过最大化层的倾斜指数目标函数竞争稀疏表示层输入。倾斜指数学习可解释为数据噪声服从高斯模型下匹配滤波器的极大似然估计。倾斜指数层的推理通过用倾斜Softmax替代批归一化实现，该机制可解释为计算每个神经元所代表的竞争信令假设的后验概率。在通过简化模型提供理论洞见后，我们在标准图像数据集上的实验表明：倾斜指数学习与推理无需数据增强即可增强对噪声及其他常见污染的鲁棒性。通过将倾斜指数与数据增强技术适当结合，可在这一系列失真条件下获得鲁棒性的进一步累积提升。