The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

翻译：近年来，训练基于Transformer的语言模型所需的计算量急剧增长。这一趋势推动了高效训练算法的研究，旨在比标准训练更快地提升训练、验证及下游任务性能。在本工作中，我们重新审视了三类此类算法：动态架构（层级堆叠、层级丢弃）、批量选择（选择性反向传播、RHO损失）以及高效优化器（Lion、Sophia）。当在固定计算预算下使用这些方法预训练BERT和T5时，我们发现，与采用完全衰减学习率的基线相比，这些方法在训练、验证及下游任务上的收益均消失殆尽。我们定义了一个评估协议，通过将所有计算时间映射到一台参考机器（我们称之为参考系统时间），使得计算可以在任意机器上进行。我们讨论了所提出协议的局限性，并公开代码以鼓励对高效训练程序进行严谨研究：https://github.com/JeanKaddour/NoTrainNoGain。