Regularization is a set of techniques that are used to improve the generalization ability of deep neural networks. In this paper, we introduce spectral batch normalization (SBN), a novel effective method to improve generalization by normalizing feature maps in the frequency (spectral) domain. The activations of residual networks without batch normalization (BN) tend to explode exponentially in the depth of the network at initialization. This leads to extremely large feature map norms even though the parameters are relatively small. These explosive dynamics can be very detrimental to learning. BN makes weight decay regularization on the scaling factors $\gamma, \beta$ approximately equivalent to an additive penalty on the norm of the feature maps, which prevents extremely large feature map norms to a certain degree. However, we show experimentally that, despite the approximate additive penalty of BN, feature maps in deep neural networks (DNNs) tend to explode at the beginning of the network and that feature maps of DNNs contain large values during the whole training. This phenomenon also occurs in a weakened form in non-residual networks. SBN addresses large feature maps by normalizing them in the frequency domain. In our experiments, we empirically show that SBN prevents exploding feature maps at initialization and large feature map values during the training. Moreover, the normalization of feature maps in the frequency domain leads to more uniform distributed frequency components. This discourages the DNNs to rely on single frequency components of feature maps. These, together with other effects of SBN, have a regularizing effect on the training of residual and non-residual networks. We show experimentally that using SBN in addition to standard regularization methods improves the performance of DNNs by a relevant margin, e.g. ResNet50 on ImageNet by 0.71%.

翻译：正则化是一类用于提升深度神经网络泛化能力的技术。本文提出频谱批量归一化（Spectral Batch Normalization, SBN），一种通过在频率（频谱）域对特征图进行归一化来改善泛化性能的新型有效方法。在初始化阶段，未使用批量归一化（BN）的残差网络激活值会随网络深度呈指数级增长，导致即使参数相对较小，特征图范数也异常庞大。这种爆炸性动力学对学习过程极为不利。BN的尺度因子$\gamma$、$\beta$上的权重衰减正则化近似等价于对特征图范数的加法惩罚，从而在一定程度上抑制了极大的特征图范数。然而，我们通过实验证明，尽管BN具有近似加法惩罚效应，深度神经网络（DNNs）中的特征图在网络浅层仍倾向于爆炸，且整个训练过程中包含较大数值。此类现象在非残差网络中以弱化形式存在。SBN通过在频域对特征图进行归一化来应对大范数特征图问题。实验表明，SBN能有效防止初始化阶段的特征图爆炸及训练过程中的大值特征图。此外，频域特征图归一化促使频率分量分布更为均匀，降低了DNNs对单一频率分量的依赖。这些效应连同SBN的其他特性，对残差网络与非残差网络的训练产生正则化效果。实验证明，在标准正则化方法基础上结合SBN可显著提升DNN性能，例如ResNet50在ImageNet上的准确率提升0.71%。