Recently, the development of pre-trained language models has brought natural language processing (NLP) tasks to the new state-of-the-art. In this paper we explore the efficiency of various pre-trained language models. We pre-train a list of transformer-based models with the same amount of text and the same training steps. The experimental results shows that the most improvement upon the origin BERT is adding the RNN-layer to capture more contextual information for the transformer-encoder layers.

翻译：最近,培训前语言模式的发展将自然语言处理(NLP)的任务带到了新的最新工艺水平上。 在本文中,我们探讨了各种培训前语言模式的效率。我们准备了一张基于变压器的模型清单,其文本与培训步骤相同。实验结果显示,对发源地BERT的最大改进是增加了RNN级,为变压器-电码层收集更多背景信息。