BERT (Bidirectional Encoder Representations from Transformers) has revolutionized the field of natural language processing through its exceptional performance on numerous tasks. Yet, the majority of researchers have mainly concentrated on enhancements related to the model structure, such as relative position embedding and more efficient attention mechanisms. Others have delved into pretraining tricks associated with Masked Language Modeling, including whole word masking. DeBERTa introduced an enhanced decoder adapted for BERT's encoder model for pretraining, proving to be highly effective. We argue that the design and research around enhanced masked language modeling decoders have been underappreciated. In this paper, we propose several designs of enhanced decoders and introduce BPDec (BERT Pretraining Decoder), a novel method for modeling training. Typically, a pretrained BERT model is fine-tuned for specific Natural Language Understanding (NLU) tasks. In our approach, we utilize the original BERT model as the encoder, making only changes to the decoder without altering the encoder. This approach does not necessitate extensive modifications to the model's architecture and can be seamlessly integrated into existing fine-tuning pipelines and services, offering an efficient and effective enhancement strategy. Compared to other methods, while we also incur a moderate training cost for the decoder during the pretraining process, our approach does not introduce additional training costs during the fine-tuning phase. We test multiple enhanced decoder structures after pretraining and evaluate their performance on the GLUE benchmark. Our results demonstrate that BPDec, having only undergone subtle refinements to the model structure during pretraining, significantly enhances model performance without escalating the inference time and serving budget.

翻译：BERT（来自Transformer的双向编码器表示）通过其在众多任务上的卓越表现革新了自然语言处理领域。然而，大多数研究者主要关注模型结构相关的改进，例如相对位置嵌入和更高效的注意力机制。其他研究者则深入探索了与掩码语言建模相关的预训练技巧，包括整词掩码。DeBERTa引入了一种适用于BERT编码器模型的增强型解码器用于预训练，并被证明非常有效。我们认为，增强型掩码语言建模解码器的设计和研究尚未得到充分重视。本文提出了多种增强型解码器的设计方案，并引入了BPDec（BERT预训练解码器）——一种新颖的训练建模方法。通常，预训练的BERT模型会针对特定的自然语言理解任务进行微调。在我们的方法中，使用原始BERT模型作为编码器，仅对解码器进行修改而不改变编码器。这种方法无需对模型架构进行大规模改动，并且可以无缝集成到现有的微调流程和服务中，提供高效且有效的增强策略。与其他方法相比，尽管我们在预训练过程中也需要为解码器付出适度的训练成本，但我们的方法在微调阶段不会引入额外的训练开销。我们测试了多种预训练后的增强型解码器结构，并在GLUE基准上评估其性能。结果表明，BPDec在预训练期间仅对模型结构进行了细微改进，即可显著提升模型性能，且不会增加推理时间和服务预算。