On-device training has become an increasingly popular approach to machine learning, enabling models to be trained directly on mobile and edge devices. However, a major challenge in this area is the limited memory available on these devices, which can severely restrict the size and complexity of the models that can be trained. In this systematic survey, we aim to explore the current state-of-the-art techniques for breaking on-device training memory walls, focusing on methods that can enable larger and more complex models to be trained on resource-constrained devices. Specifically, we first analyze the key factors that contribute to the phenomenon of memory walls encountered during on-device training. Then, we present a comprehensive literature review of on-device training, which addresses the issue of memory limitations. Finally, we summarize on-device training and highlight the open problems for future research. By providing a comprehensive overview of these techniques and their effectiveness in breaking memory walls, we hope to help researchers and practitioners in this field navigate the rapidly evolving landscape of on-device training.

翻译：端侧训练已成为机器学习领域日益流行的范式，支持在移动和边缘设备上直接完成模型训练。然而，该领域面临的核心挑战在于设备有限的内存资源，这严重制约了可训练模型的规模与复杂度。本系统性综述旨在探索突破端侧训练内存墙的现有前沿技术，重点聚焦于在资源受限设备上实现更大规模、更高复杂度模型训练的方法。具体而言，我们首先分析了导致端侧训练中内存墙现象的关键因素。随后，对涉及内存限制问题的端侧训练研究进行了全面文献梳理。最后，我们总结了端侧训练现状，并指出了未来研究中的开放性问题。通过系统概述这些技术及其在突破内存墙方面的有效性，我们期望为相关领域的研究人员与从业者提供导航，助其把握端侧训练这一快速发展的领域。