Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.

翻译：近年来Transformer模型的发展对计算资源提出了巨大需求，凸显了开发高效训练技术的重要性——通过高效利用计算与内存资源，实现更快速、更低成本且更高精度的Transformer训练。本文首次系统综述了Transformer高效训练领域的研究进展，涵盖加速算法与硬件方面的最新成果，并重点关注前者。我们分析并比较了在训练过程中通过节省中间张量计算与内存成本的方法，以及硬件/算法协同设计技术。最后讨论了当前面临的挑战及未来具有潜力的研究方向。