Large-scale deep neural networks consume expensive training costs, but the training results in less-interpretable weight matrices constructing the networks. Here, we propose a mode decomposition learning that can interpret the weight matrices as a hierarchy of latent modes. These modes are akin to patterns in physics studies of memory networks, but the least number of modes increases only logarithmically with the network width, and becomes even a constant when the width further grows. The mode decomposition learning not only saves a significant large amount of training costs, but also explains the network performance with the leading modes, displaying a striking piecewise power-law behavior. The modes specify a progressively compact latent space across the network hierarchy, making a more disentangled subspaces compared to standard training. Our mode decomposition learning is also studied in an analytic on-line learning setting, which reveals multi-stage of learning dynamics with a continuous specialization of hidden nodes. Therefore, the proposed mode decomposition learning points to a cheap and interpretable route towards the magical deep learning.

翻译：大规模深度神经网络消耗了昂贵的训练成本，但训练结果构建出可解释性较差的权重矩阵。本文提出一种模态分解学习方法，可将权重矩阵解释为层级潜在模式。这些模式类似于记忆网络物理研究中的模式，但最少模式数量仅随网络宽度呈对数增长，当宽度进一步增大时甚至趋于常数。该模态分解学习不仅显著节省大量训练成本，还能通过主导模式解释网络性能，展现出惊人的分段幂律行为。这些模式在网络层级中逐步压缩潜在空间，相较于标准训练方法形成更解耦的子空间。我们在解析在线学习框架下研究了该模态分解学习，揭示了具备隐节点持续特化的多阶段学习动力学特征。因此，本文提出的模态分解学习为通向神秘深度学习提供了一条廉价且可解释的路径。