In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.

翻译：本文论证了深度学习科学理论正在形成的观点。这里所说的理论，是指能够表征神经网络训练过程、隐藏表征、最终权重及性能等重要属性与统计特性的理论体系。我们梳理了当前深度学习理论研究的五大主要方向，这些方向共同指向该理论的形成：(a) 可解的理想化场景，为真实系统的学习动力学提供直观认识；(b) 易于处理的理论极限，揭示基础学习现象的深刻洞察；(c) 捕捉重要宏观可观测量的简洁数学定律；(d) 超参数理论，使其与训练过程的其他部分解耦，从而简化剩余系统；(e) 跨系统与场景共存的普适行为，明确哪些现象需要解释。综合来看，这些研究方向具备共同特征：关注训练过程的动力学；主要致力于描述粗粒度的聚合统计量；强调可证伪的定量预测。我们认为这一新兴理论最宜被视为学习过程的力学，并提议将其命名为"学习力学"。我们探讨了该力学视角与其他构建深度学习理论方法（如统计视角与信息论视角）的关系，特别预见学习力学与机械可解释性之间将形成共生关系。本文亦回顾并回应了关于基础理论不可行或不重要的常见论点，最后描绘了学习力学的重要开放方向，并为初学者提供建议。更多导引材料、视角及开放问题可参阅learningmechanics.pub。