Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt{[MASK]}$ symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens. Motivated by the identified issue, we propose MAE-LM, which pretrains the Masked Autoencoder architecture with MLM where $\texttt{[MASK]}$ tokens are excluded from the encoder. Empirically, we show that MAE-LM improves the utilization of model dimensions for real token representations, and MAE-LM consistently outperforms MLM-pretrained models across different pretraining settings and model sizes when fine-tuned on the GLUE and SQuAD benchmarks.

翻译：掩码语言建模（Masked Language Modeling, MLM）因其简洁性和有效性，已成为预训练双向文本编码器的最主要方法之一。MLM的一个显著问题是，特殊的$\texttt{[MASK]}$符号仅在预训练阶段出现而微调阶段不存在，导致预训练数据与下游数据之间存在差异。本文对这种差异的后果提出了新的视角：我们通过实验和理论证明，MLM预训练会将部分模型维度专门用于表示$\texttt{[MASK]}$标记，从而导致真实标记的表示缺陷，并限制了预训练模型在适应不含$\texttt{[MASK]}$标记的下游数据时的表达能力。基于识别出的问题，我们提出MAE-LM方法，该方法采用掩码自编码器架构进行MLM预训练，且编码器中不包含$\texttt{[MASK]}$标记。实验表明，MAE-LM改善了模型维度对真实标记表示的利用效率，在GLUE和SQuAD基准测试中，无论预训练设置或模型规模如何，MAE-LM在微调后均持续优于MLM预训练模型。