The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

翻译：近年来，训练基于Transformer的语言模型所需的计算量急剧增长。这一趋势推动了高效训练算法的研究，旨在比标准训练更快地提升训练、验证及下游性能。本研究对三类此类算法进行了重访：动态架构（层次堆叠、层次丢弃）、批量选择（选择性反向传播、RHO损失）以及高效优化器（Lion、Sophia）。在固定计算预算下使用这些方法预训练BERT和T5时，我们发现与采用完全衰减学习率的基线相比，其在训练、验证和下游性能上的提升均不复存在。我们定义了一个评估协议，通过将所有计算时间映射到参考机器（称为参考系统时间），使得计算可在任意机器上执行。我们讨论了所提协议的局限性，并公开代码以鼓励高效训练流程的严谨研究：https://github.com/JeanKaddour/NoTrainNoGain。