Many data symmetries can be described in terms of group equivariance and the most common way of encoding group equivariances in neural networks is by building linear layers that are group equivariant. In this work we investigate whether equivariance of a network implies that all layers are equivariant. On the theoretical side we find cases where equivariance implies layerwise equivariance, but also demonstrate that this is not the case generally. Nevertheless, we conjecture that CNNs that are trained to be equivariant will exhibit layerwise equivariance and explain how this conjecture is a weaker version of the recent permutation conjecture by Entezari et al. [2022]. We perform quantitative experiments with VGG-nets on CIFAR10 and qualitative experiments with ResNets on ImageNet to illustrate and support our theoretical findings. These experiments are not only of interest for understanding how group equivariance is encoded in ReLU-networks, but they also give a new perspective on Entezari et al.'s permutation conjecture as we find that it is typically easier to merge a network with a group-transformed version of itself than merging two different networks.

翻译：许多数据对称性可以用群等变性来描述，而在神经网络中编码群等变性的最常见方式是构建具有群等变性的线性层。本研究探讨了网络的等变性是否意味着所有层都具有等变性。在理论层面，我们发现了等变性蕴含逐层等变性的某些情况，但也证明这一结论并非普遍成立。尽管如此，我们推测经过训练具备等变性的卷积神经网络将展现逐层等变性，并阐释了这一推测可视为Entezari等人[2022]最新排列猜想的一个较弱版本。我们通过CIFAR10数据集上VGG网络的定量实验和ImageNet数据集上ResNet网络的定性实验，阐明并支持了理论发现。这些实验不仅有助于理解ReLU网络如何编码群等变性，还为解读Entezari等人的排列猜想提供了新视角——我们发现将网络与其群变换版本融合通常比融合两个不同网络更为容易。