Masked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training. However, the model may receive complicated impact from pre-training status, which changes accordingly as training time goes on. In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings. We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks. Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pre-training.

翻译：掩码语言建模（MLM）作为去噪目标已广泛应用于语言模型预训练（PrLMs）。现有PrLMs通常采用随机词元掩码策略，即以固定掩码比例在整个训练过程中以均等概率掩码不同内容。然而，模型可能受到随训练时间动态变化的预训练状态的复杂影响。本文证明这种掩码比例与掩码内容均保持时间不变性的MLM设置难以获得最优结果，这促使我们探索时变MLM设置的影响。我们提出两种调度掩码方法，能在不同训练阶段自适应调整掩码比例与掩码内容，从而在下游任务上验证了预训练效率与有效性的提升。本文首次系统研究了掩码比例与内容的时变策略，为理解掩码比例与掩码内容如何影响MLM预训练提供了更深入的见解。