We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth' hyper-parameter $\tilde{L}$. In the infinite depth limit, we study 'representation geodesics' $A_{p}$: continuous paths in representation space (similar to NeuralODEs) from input $p=0$ to output $p=1$ that minimize the parameter norm of the network. We give a Lagrangian and Hamiltonian reformulation, which highlight the importance of two terms: a kinetic energy which favors small layer derivatives $\partial_{p}A_{p}$ and a potential energy that favors low-dimensional representations, as measured by the 'Cost of Identity'. The balance between these two forces offers an intuitive understanding of feature learning in ResNets. We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work: for large $\tilde{L}$ the potential energy dominates and leads to a separation of timescales, where the representation jumps rapidly from the high dimensional inputs to a low-dimensional representation, move slowly inside the space of low-dimensional representations, before jumping back to the potentially high-dimensional outputs. Inspired by this phenomenon, we train with an adaptive layer step-size to adapt to the separation of timescales.

翻译：我们研究泄漏残差网络（Leaky ResNets），该网络通过'有效深度'超参数$\tilde{L}$在残差网络（ResNets）与全连接网络之间插值。在无限深度极限下，我们研究'表示测地线'$A_{p}$：表示空间中从输入$p=0$到输出$p=1$的连续路径（类似于神经常微分方程），该路径最小化网络的参数范数。我们提出拉格朗日与哈密顿重构，突显两项关键作用：倾向于小层导数$\partial_{p}A_{p}$的动能，以及通过'恒等代价'衡量、倾向于低维表示势能。这两力平衡为残差网络中的特征学习提供了直观理解。我们利用该直觉解释先前研究中观察到的瓶颈结构涌现现象：当$\tilde{L}$较大时，势能主导并导致时间尺度分离——表示从高维输入快速跃迁至低维表示，在低维表示空间缓慢移动，再跳回可能的高维输出。受此现象启发，我们采用自适应层步长训练以适应时间尺度分离。