Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training. These methods simply consider two passages from the same document to be relevant, without taking into account the possibility of weakly correlated pairs. Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue. Query-as-context pre-training assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervised pre-training. The pre-trained models are evaluated on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. Experimental results show that query-as-context pre-training brings considerable gains and meanwhile speeds up training, demonstrating its effectiveness and efficiency. Our code will be available at https://github.com/caskcsg/ir/tree/main/cotmae-qc .

翻译：近期，研究者通过使用上下文监督预训练方法提升了密集段落检索的性能。这类方法简单地将同一文档中的两个段落视为相关，未考虑弱相关对的可能性。为此，本文提出查询即上下文预训练技术——一种简单有效的预训练方法以缓解该问题。该技术假设从段落中衍生的查询更可能与对应段落相关，从而形成段落-查询对，并将这些对用于对比式或生成式上下文监督预训练。在大型段落检索基准与跨领域零样本基准上的评估表明，查询即上下文预训练不仅能带来显著的性能提升，同时可加速训练过程，证明了其有效性与高效性。我们的代码将发布在 https://github.com/caskcsg/ir/tree/main/cotmae-qc 。