Not all learnable parameters (e.g., weights) contribute equally to a neural network's decision function. In fact, entire layers' parameters can sometimes be reset to random values with little to no impact on the model's decisions. We revisit earlier studies that examined how architecture and task complexity influence this phenomenon and ask: is this phenomenon also affected by how we train the model? We conducted experimental evaluations on a diverse set of ImageNet-1k classification models to explore this, keeping the architecture and training data constant but varying the training pipeline. Our findings reveal that the training method strongly influences which layers become critical to the decision function for a given task. For example, improved training regimes and self-supervised training increase the importance of early layers while significantly under-utilizing deeper layers. In contrast, methods such as adversarial training display an opposite trend. Our preliminary results extend previous findings, offering a more nuanced understanding of the inner mechanics of neural networks. Code: https://github.com/paulgavrikov/layer_criticality

翻译：并非所有可学习参数（如权重）对神经网络的决策函数贡献均等。事实上，有时将整个层的参数重置为随机值，对模型的决策几乎不产生影响。我们重新审视了先前关于架构与任务复杂度如何影响此现象的研究，并进一步提出：训练模型的方式是否也会影响该现象？为此，我们在多种ImageNet-1k分类模型上进行了实验评估，保持架构和训练数据不变，仅改变训练流程。研究结果表明，训练方法强烈影响哪些层对特定任务的决策函数至关重要。例如，改进的训练方案和自监督训练会提升浅层的重要性，同时显著降低深层网络的利用率。相反，对抗训练等方法则呈现相反的趋势。我们的初步结果扩展了先前的研究发现，为神经网络内部机制提供了更细致的理解。代码：https://github.com/paulgavrikov/layer_criticality