State-of-the-art techniques for enhancing robustness of deep networks mostly rely on empirical risk minimization with suitable data augmentation. In this paper, we propose a complementary approach motivated by communication theory, aimed at enhancing the signal-to-noise ratio at the output of a neural network layer via neural competition during learning and inference. In addition to standard empirical risk minimization, neurons compete to sparsely represent layer inputs by maximization of a tilted exponential (TEXP) objective function for the layer. TEXP learning can be interpreted as maximum likelihood estimation of matched filters under a Gaussian model for data noise. Inference in a TEXP layer is accomplished by replacing batch norm by a tilted softmax, which can be interpreted as computation of posterior probabilities for the competing signaling hypotheses represented by each neuron. After providing insights via simplified models, we show, by experimentation on standard image datasets, that TEXP learning and inference enhances robustness against noise and other common corruptions, without requiring data augmentation. Further cumulative gains in robustness against this array of distortions can be obtained by appropriately combining TEXP with data augmentation techniques. The code for all our experiments is available at https://github.com/bhagyapuranik/texp_for_robustness.

翻译：当前提升深度网络鲁棒性的主流技术主要依赖于结合适当数据增强的经验风险最小化。本文提出了一种受通信理论启发的互补方法，旨在通过学习与推理过程中的神经竞争，增强神经网络层输出的信噪比。在标准经验风险最小化基础上，神经元通过最大化层级的倾斜指数目标函数来竞争稀疏表示层输入。倾斜指数学习可被解释为在数据噪声的高斯模型假设下对匹配滤波器的最大似然估计。倾斜指数层的推理通过用倾斜Softmax替代批归一化实现，该操作可被解释为对每个神经元所代表的竞争性信号假设的后验概率计算。通过简化模型的理论分析，我们基于标准图像数据集的实验表明，倾斜指数学习与推理无需数据增强即可提升对噪声及其他常见污染的鲁棒性。进一步，将倾斜指数与数据增强技术适当结合，可累积提升对此类多重失真的鲁棒性。所有实验代码已开源至https://github.com/bhagyapuranik/texp_for_robustness。