The utilization of residual learning has become widespread in deep and scalable neural nets. However, the fundamental principles that contribute to the success of residual learning remain elusive, thus hindering effective training of plain nets with depth scalability. In this paper, we peek behind the curtains of residual learning by uncovering the "dissipating inputs" phenomenon that leads to convergence failure in plain neural nets: the input is gradually compromised through plain layers due to non-linearities, resulting in challenges of learning feature representations. We theoretically demonstrate how plain neural nets degenerate the input to random noise and emphasize the significance of a residual connection that maintains a better lower bound of surviving neurons as a solution. With our theoretical discoveries, we propose "The Plain Neural Net Hypothesis" (PNNH) that identifies the internal path across non-linear layers as the most critical part in residual learning, and establishes a paradigm to support the training of deep plain neural nets devoid of residual connections. We thoroughly evaluate PNNH-enabled CNN architectures and Transformers on popular vision benchmarks, showing on-par accuracy, up to 0.3% higher training throughput, and 2x better parameter efficiency compared to ResNets and vision Transformers.

翻译：残差学习已广泛应用于深度可扩展神经网络。然而，残差学习成功的基本原理仍不明确，这阻碍了普通网络实现深度可扩展的有效训练。本文通过揭示导致普通神经网络收敛失败的"输入消散"现象，窥探残差学习幕后的机制：由于非线性作用，输入通过普通层时逐渐退化，导致学习特征表示面临挑战。我们从理论上论证了普通神经网络如何将输入退化为随机噪声，并强调了残差连接作为解决方案在维持存活神经元更优下界方面的重要意义。基于理论发现，我们提出"普通神经网络假说"（PNNH），该假说将跨非线性层的内部路径识别为残差学习中最关键的部分，并建立了一种支持无残差连接的深度普通网络训练范式。我们在主流视觉基准上全面评估了PNNH赋能的CNN架构和Transformer，结果表明其与ResNet和视觉Transformer相比，准确率持平，训练吞吐量提升高达0.3%，参数效率提升2倍。