Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically schedule the masking rate throughout training. We find that linearly decreasing the masking rate over the course of pretraining improves average GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT-large, respectively, compared to fixed rate baselines. These gains come from exposure to both high and low masking rate regimes, providing benefits from both settings. Our results demonstrate that masking rate scheduling is a simple way to improve the quality of masked language models, achieving up to a 1.89x speedup in pretraining for BERT-base as well as a Pareto improvement for BERT-large.

翻译：大多数采用掩码语言建模（MLM）目标训练的Transformer研究沿用原始BERT模型的固定15%掩码率。我们提出在训练过程中动态调度掩码率的方案。研究发现，与固定掩码率基线相比，在预训练期间线性降低掩码率可使BERT-base和BERT-large在GLUE基准上的平均准确率分别提升0.46%和0.25%。这些性能提升源于模型同时受益于高掩码率与低掩码率两种训练模式。实验结果表明，掩码率调度作为提升掩码语言模型质量的简易方法，可使BERT-base的预训练速度提升1.89倍，并对BERT-large实现帕累托改进。