The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

翻译：近年来，训练基于Transformer的语言模型所需的计算量急剧增长。这一趋势促使了高效训练算法的研究，其目标在于相比标准训练，能更快提升训练、验证及下游任务的性能。在本工作中，我们重新审视了三类此类算法：动态架构（层堆叠、层丢弃）、批量选择（选择性反向传播、RHO损失）以及高效优化器（Lion、Sophia）。在采用固定计算预算并使用此类方法预训练BERT和T5时，我们发现，相较于采用完全衰减学习率的基线方法，其在训练、验证及下游任务上的性能增益消失殆尽。我们定义了一个评估协议，通过将所有计算时间映射到一台参考机器（我们称之为参考系统时间），使得计算可以在任意机器上进行。我们讨论了所提协议的局限性，并公开了代码以鼓励对高效训练程序进行严谨的研究：https://github.com/JeanKaddour/NoTrainNoGain。