We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques. Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. This design reduces the intrinsic cross-sample dependency of BN and enhances the generalization of the teacher. EMAN improves strong baselines for self-supervised learning by 4-6/1-2 points and semi-supervised learning by about 7/2 points, when 1%/10% supervised labels are available on ImageNet. These improvements are consistent across methods, network architectures, training duration, and datasets, demonstrating the general effectiveness of this technique. The code is available at https://github.com/amazon-research/exponential-moving-average-normalization.

翻译：我们为批次正常化(BN)提供了一个称为指数移动平均正常化(EMAN)的插件替换,这改善了现有学生-教师自学和半监督学习技术的绩效。与标准BN(每批中计算统计数据)不同的是,教师使用的EMAN(EMAN)通过学生BN统计数字的指数移动平均数更新其统计数据。这一设计减少了BN固有的交叉依赖性并加强了教师的普及性。EMAN通过4-6/1-2点和半监督学习改进了自我监督学习的强大基线,在图像网络上提供了1%/10%的监管标签。这些改进在方法、网络结构、培训期限和数据集方面是一致的,显示了这一技术的总体效力。该代码可在https://github.com/amazon-research/exconential-moviving-平均正常化网站查阅。