To better support information retrieval tasks such as web search and open-domain question answering, growing effort is made to develop retrieval-oriented language models, e.g., RetroMAE and many others. Most of the existing works focus on improving the semantic representation capability for the contextualized embedding of the [CLS] token. However, recent study shows that the ordinary tokens besides [CLS] may provide extra information, which help to produce a better representation effect. As such, it's necessary to extend the current methods where all contextualized embeddings can be jointly pre-trained for the retrieval tasks. In this work, we propose a novel pre-training method called Duplex Masked Auto-Encoder, a.k.a. DupMAE. It is designed to improve the quality of semantic representation where all contextualized embeddings of the pre-trained model can be leveraged. It takes advantage of two complementary auto-encoding tasks: one reconstructs the input sentence on top of the [CLS] embedding; the other one predicts the bag-of-words feature of the input sentence based on the ordinary tokens' embeddings. The two tasks are jointly conducted to train a unified encoder, where the whole contextualized embeddings are aggregated in a compact way to produce the final semantic representation. DupMAE is simple but empirically competitive: it substantially improves the pre-trained model's representation capability and transferability, where superior retrieval performances can be achieved on popular benchmarks, like MS MARCO and BEIR.

翻译：为更好地支持网络搜索和开放域问答等信息检索任务，学界日益致力于开发面向检索的语言模型（如RetroMAE等）。现有工作大多聚焦于提升[CLS]标记的上下文嵌入语义表征能力。然而，近期研究表明，除[CLS]外的普通标记可能提供额外信息，有助于产生更优的表征效果。因此，有必要扩展现有方法，使所有上下文嵌入能够联合预训练以服务于检索任务。本文提出一种名为双重掩码自编码器（DupMAE）的新型预训练方法，旨在提升预训练模型所有上下文嵌入可被利用时的语义表征质量。该方法充分利用两种互补的自编码任务：其一基于[CLS]嵌入重构输入句子，其二基于普通标记的嵌入预测输入句子的词袋特征。两项任务联合执行以训练统一编码器，使所有上下文嵌入以紧凑方式聚合，生成最终语义表征。DupMAE简洁而具有实证竞争力：它能显著提升预训练模型的表征能力与迁移性，在MS MARCO和BEIR等主流基准测试中均取得优异检索性能。