State-of-the-art techniques for enhancing robustness of deep networks mostly rely on empirical risk minimization with suitable data augmentation. In this paper, we propose a complementary approach motivated by communication theory, aimed at enhancing the signal-to-noise ratio at the output of a neural network layer via neural competition during learning and inference. In addition to minimization of a standard end-to-end cost, neurons compete to sparsely represent layer inputs by maximization of a tilted exponential (TEXP) objective function for the layer. TEXP learning can be interpreted as maximum likelihood estimation of matched filters under a Gaussian model for data noise. Inference in a TEXP layer is accomplished by replacing batch norm by a tilted softmax, which can be interpreted as computation of posterior probabilities for the competing signaling hypotheses represented by each neuron. After providing insights via simplified models, we show, by experimentation on standard image datasets, that TEXP learning and inference enhances robustness against noise and other common corruptions, without requiring data augmentation. Further cumulative gains in robustness against this array of distortions can be obtained by appropriately combining TEXP with data augmentation techniques.

翻译：增强深度网络鲁棒性的最先进技术主要依赖于结合适当数据增强的经验风险最小化。本文提出一种受通信理论启发的互补方法，旨在通过学习与推理过程中的神经竞争提升神经网络层输出的信噪比。在最小化标准端到端代价函数的同时，神经元通过最大化该层的倾斜指数目标函数，以稀疏方式表示层输入。倾斜指数学习可解释为在数据噪声服从高斯模型条件下对匹配滤波器进行最大似然估计。倾斜指数层的推理通过用倾斜softmax替代批归一化实现，该操作可解释为对每个神经元所代表的竞争信号假设进行后验概率计算。在通过简化模型获得理论见解后，我们在标准图像数据集上的实验表明，倾斜指数学习与推理无需数据增强即可提升对噪声及其他常见扰动的鲁棒性。通过将倾斜指数与数据增强技术适当结合，可进一步累积提升针对此类失真的鲁棒性。