Masked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training. However, the model may receive complicated impact from pre-training status, which changes accordingly as training time goes on. In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings. We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks. Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pre-training.

翻译：掩码语言建模（MLM）作为预训练语言模型（PrLMs）中广泛使用的去噪目标，现有PrLMs通常采用随机标记掩码策略，即在训练全过程中以固定掩码比例对不同内容等概率进行掩码。然而，随着训练时间的推移，模型可能受到预训练状态的复杂影响，这种影响会动态变化。本文研究表明，这种在掩码比例和掩码内容上保持时间不变的MLM设置难以达到最优效果，这促使我们探索时变MLM设置的影响。我们提出两种计划性掩码方法，能在不同训练阶段自适应调整掩码比例和掩码内容，从而在下游任务中验证了预训练效率与效果的双重提升。本研究首次系统探讨了掩码比例与内容的时间动态策略，为理解掩码比例和掩码内容如何影响MLM预训练提供了新视角。