BERT (Bidirectional Encoder Representations from Transformers) has revolutionized the field of natural language processing through its exceptional performance on numerous tasks. Yet, the majority of researchers have mainly concentrated on enhancements related to the model structure, such as relative position embedding and more efficient attention mechanisms. Others have delved into pretraining tricks associated with Masked Language Modeling, including whole word masking. DeBERTa introduced an enhanced decoder adapted for BERT's encoder model for pretraining, proving to be highly effective. We argue that the design and research around enhanced masked language modeling decoders have been underappreciated. In this paper, we propose several designs of enhanced decoders and introduce DrBERT (Decoder-refined BERT), a novel method for modeling training. Typically, a pretrained BERT model is fine-tuned for specific Natural Language Understanding (NLU) tasks. In our approach, we utilize the original BERT model as the encoder, making only changes to the decoder without altering the encoder. This approach does not necessitate extensive modifications to the model's architecture and can be seamlessly integrated into existing fine-tuning pipelines and services, offering an efficient and effective enhancement strategy. Compared to other methods, while we also incur a moderate training cost for the decoder during the pretraining process, our approach does not introduce additional training costs during the fine-tuning phase. We test multiple enhanced decoder structures after pretraining and evaluate their performance on the GLUE benchmark. Our results demonstrate that DrBERT, having only undergone subtle refinements to the model structure during pretraining, significantly enhances model performance without escalating the inference time and serving budget.

翻译：BERT（来自Transformer的双向编码器表示）通过其在众多任务上的卓越表现，革新了自然语言处理领域。然而，大多数研究者主要关注模型结构的改进，如相对位置嵌入和更高效的注意力机制；另一些研究者则深入探索与掩码语言建模相关的预训练技巧，包括整词掩码。DeBERTa引入了一种针对BERT编码器模型进行预训练改进的解码器，被证明非常有效。我们认为，关于增强型掩码语言建模解码器的设计与研究一直未得到充分重视。本文提出了多种增强解码器的设计方案，并介绍了DrBERT（解码器优化的BERT），这是一种新颖的建模训练方法。通常，预训练的BERT模型会针对特定的自然语言理解（NLU）任务进行微调。在我们的方法中，我们使用原始BERT模型作为编码器，仅对解码器进行修改而不改变编码器。这种方法无需对模型架构进行大规模改动，可无缝集成到现有的微调流程和服务中，提供了一种高效且有效的增强策略。与其他方法相比，尽管我们在预训练过程中也需要承担适度的解码器训练成本，但我们的方法在微调阶段不会引入额外的训练成本。我们对预训练后的多种增强解码器结构进行了测试，并在GLUE基准上评估其性能。结果表明，DrBERT虽然在预训练期间仅对模型结构进行了精细调整，却能显著提升模型性能，且不会增加推理时间和服务预算。