The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

翻译：近年来，训练基于Transformer的语言模型所需的计算成本急剧增加。这一趋势推动了高效训练算法的研究，旨在比标准训练更快地提升训练、验证及下游任务性能。本研究重新审视了三类此类算法：动态架构（层堆叠、层丢弃）、批量选择（选择性反向传播、RHO损失）以及高效优化器（Lion、Sophia）。在采用固定计算预算预训练BERT和T5时，我们发现，相较于使用完全衰减学习率的基线方法，这些算法在训练、验证及下游任务上的性能提升均不复存在。我们定义了一套评估协议，通过将所有计算时间映射到参考机器（称为参考系统时间），从而允许在任意机器上执行计算。我们讨论了所提出协议的局限性，并公开代码以鼓励对高效训练流程进行严谨研究：https://github.com/JeanKaddour/NoTrainNoGain。