Growing techniques have been emerging to improve the performance of passage retrieval. As an effective representation bottleneck pretraining technique, the contextual masked auto-encoder utilizes contextual embedding to assist in the reconstruction of passages. However, it only uses a single auto-encoding pre-task for dense representation pre-training. This study brings multi-view modeling to the contextual masked auto-encoder. Firstly, multi-view representation utilizes both dense and sparse vectors as multi-view representations, aiming to capture sentence semantics from different aspects. Moreover, multiview decoding paradigm utilizes both autoencoding and auto-regressive decoders in representation bottleneck pre-training, aiming to provide both reconstructive and generative signals for better contextual representation pretraining. We refer to this multi-view pretraining method as CoT-MAE v2. Through extensive experiments, we show that CoT-MAE v2 is effective and robust on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks.

翻译：面向段落检索性能提升的技术不断涌现。作为有效的表示瓶颈预训练技术，上下文掩码自编码器利用上下文嵌入辅助段落重建，但仅依赖单一自编码预任务进行密集表示预训练。本研究将多视角建模引入上下文掩码自编码器：首先，多视角表示同时利用密集向量和稀疏向量作为多视角表征，旨在从不同层面捕捉句子语义；其次，多视角解码范式在表示瓶颈预训练中同时使用自编码和自回归解码器，旨在通过重建信号与生成信号的双重驱动，优化上下文表示预训练效果。我们将这种多视角预训练方法命名为CoT-MAE v2。大量实验表明，CoT-MAE v2在大规模段落检索基准和跨领域零样本基准上均展现出有效性与鲁棒性。