Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy. However, random masking does not consider the importance of the different words in the sentence meaning, where some of them are more worthy to be predicted. Therefore, various masking strategies (e.g., entity-level masking) are proposed, but most of them require expensive prior knowledge and generally train from scratch without reusing existing model weights. In this paper, we present Self-Evolution learning (SE), a simple and effective token masking and learning method to fully and wisely exploit the knowledge from data. SE focuses on learning the informative yet under-explored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach. Experiments on 10 tasks show that our SE brings consistent and significant improvements (+1.43~2.12 average scores) upon different PLMs. In-depth analyses demonstrate that SE improves linguistic knowledge learning and generalization.

翻译：掩码语言建模作为判别式语言模型（如BERT）预训练中的主流方法，通常采用随机掩码策略。然而随机掩码未能考虑词语在句子语义中的重要性差异——部分词语更值得被预测。为此，学界提出了多种掩码策略（如实体级掩码），但多数方案需要高昂的先验知识成本，且通常需从零开始训练而无法复用现有模型权重。本文提出自演化学习（Self-Evolution Learning, SE），一种简洁有效的词元掩码与学习方法，可充分且智能地挖掘数据中的知识。SE聚焦于学习信息丰富但挖掘不足的词元，并通过引入创新的词元特异标签平滑（Token-specific Label Smoothing）方法实现训练的自适应正则化。在10项任务上的实验表明，SE在不同预训练语言模型上均实现了持续显著的性能提升（平均得分提高1.43~2.12）。深入分析显示，SE可有效增强语言学知识学习与模型泛化能力。