ELECTRA pre-trains language models by detecting tokens in a sequence that have been replaced by an auxiliary model. Although ELECTRA offers a significant boost in efficiency, its potential is constrained by the training cost brought by the auxiliary model. Notably, this model, which is jointly trained with the main model, only serves to assist the training of the main model and is discarded post-training. This results in a substantial amount of training cost being expended in vain. To mitigate this issue, we propose Fast-ELECTRA, which leverages an existing language model as the auxiliary model. To construct a learning curriculum for the main model, we smooth its output distribution via temperature scaling following a descending schedule. Our approach rivals the performance of state-of-the-art ELECTRA-style pre-training methods, while significantly eliminating the computation and memory cost brought by the joint training of the auxiliary model. Our method also reduces the sensitivity to hyper-parameters and enhances the pre-training stability.

翻译：ELECTRA通过检测序列中被辅助模型替换的标记来预训练语言模型。尽管ELECTRA显著提升了效率，但其潜力受限于辅助模型带来的训练成本。值得注意的是，该模型与主模型联合训练，仅用于辅助主模型的训练过程，并在训练完成后被丢弃，这导致大量训练成本被无效消耗。为解决这一问题，我们提出Fast-ELECTRA，该方法利用现有语言模型作为辅助模型。为了为主模型构建学习课程，我们采用温度缩放技术，通过递减调度策略平滑其输出分布。我们的方法在性能上可与最先进的ELECTRA式预训练方法相媲美，同时显著消除了辅助模型联合训练带来的计算与内存开销。此外，该方法降低了对超参数的敏感性，并增强了预训练的稳定性。