Normalization layers and activation functions are critical components in deep neural networks that frequently co-locate with each other. Instead of designing them separately, we unify them into a single computation graph, and evolve its structure starting from low-level primitives. Our layer search algorithm leads to the discovery of EvoNorms, a set of new normalization-activation layers that go beyond existing design patterns. Several of these layers enjoy the property of being independent from the batch statistics. Our experiments show that EvoNorms not only excel on a variety of image classification models including ResNets, MobileNets and EfficientNets, but also transfer well to Mask R-CNN for instance segmentation and BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers by a significant margin in many cases.

翻译：正常化层和激活功能是深层神经网络的关键组成部分,这些网络经常相互合用。 我们不是单独设计它们,而是将它们合并成一个单一的计算图,并从低层次原始人开始演变其结构。 我们的层搜索算法导致发现一套超越现有设计模式的新的正常化活动层EvoNorms。 其中一些层享有独立于批量统计的特性。 我们的实验显示, EvoNorms不仅在各种图像分类模型上非常出色,包括ResNets、移动网络和高效网络,而且向Mask R-CNN(例如分解)和BigGAN(例如图像合成、优于BatchNorm)和GroupNorm(在许多情况下以显著的差值)转移到图像系统R-CNN(例如分解)和BigGAN(例如图像合成、优于BatchNorm和Group Norm)的层。