On-device training has become an increasingly popular approach to machine learning, enabling models to be trained directly on mobile and edge devices. However, a major challenge in this area is the limited memory available on these devices, which can severely restrict the size and complexity of the models that can be trained. In this systematic survey, we aim to explore the current state-of-the-art techniques for breaking on-device training memory walls, focusing on methods that can enable larger and more complex models to be trained on resource-constrained devices. Specifically, we first analyze the key factors that contribute to the phenomenon of memory walls encountered during on-device training. Then, we present a comprehensive literature review of on-device training, which addresses the issue of memory limitations. Finally, we summarize on-device training and highlight the open problems for future research. By providing a comprehensive overview of these techniques and their effectiveness in breaking memory walls, we hope to help researchers and practitioners in this field navigate the rapidly evolving landscape of on-device training.

翻译：设备端训练已成为机器学习中日益流行的方法，使得模型能够直接在移动和边缘设备上进行训练。然而，该领域的一大挑战在于这些设备上有限的内存，这会严重限制可训练模型的规模和复杂度。在本系统性综述中，我们旨在探索打破设备端训练内存墙的当前最先进技术，重点关注能够在资源受限设备上训练更大、更复杂模型的方法。具体而言，我们首先分析了导致设备端训练过程中出现内存墙现象的关键因素。随后，我们针对设备端训练中内存限制问题进行了全面的文献综述。最后，我们对设备端训练进行了总结，并指出了未来研究的开放性问题。通过全面概述这些技术及其在打破内存墙方面的有效性，我们希望能帮助该领域的研究人员与实践者驾驭快速演进的设备端训练领域。