Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.

翻译：近年来Transformer模型的进展对计算资源提出了巨大需求，凸显了开发高效训练技术的重要性——通过高效利用计算和内存资源，实现更快、更低成本且更高精度的Transformer训练。本综述首次系统梳理了Transformer高效训练的研究进展，涵盖加速算法与硬件领域的最新成果，并重点聚焦前者。我们分析比较了训练过程中降低中间张量计算与内存开销的方法，以及硬件/算法协同设计技术。最后讨论了当前面临的挑战与未来具有潜力的研究方向。