While in-context learning is well-studied with decoder-only language models (LLMs), its utility for encoder-only models remains underexplored. We study in-context learning for encoder-only models for text retrieval tasks. Can incorporating in-context examples (query-document pairs) to the target query enhance retriever performance? Our approach, RARe, finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This approach achieves performance gains of up to +2.72% nDCG across open-domain retrieval datasets (BeIR, RAR-b) compared to using the target query only as an input. In particular, we find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. We further provide analysis on the design choices of in-context example augmentation for retrievers and lay the foundation for future work.

翻译：尽管上下文学习在仅解码器语言模型（LLMs）中已得到充分研究，但其在仅编码器模型中的应用仍待深入探索。本文研究了仅编码器模型在文本检索任务中的上下文学习能力：通过将上下文示例（查询-文档对）融入目标查询，能否提升检索器的性能？我们提出的方法RARe，通过微调预训练模型，并引入与目标查询语义相似的上下文示例，实现了检索性能的提升。在开放域检索数据集（BeIR、RAR-b）上，相较于仅使用目标查询作为输入的方法，RARe取得了最高+2.72% nDCG的性能增益。特别地，我们发现RARe展现出更强的跨领域泛化能力，这与LLMs中上下文学习的表现相似。我们进一步分析了检索器中上下文示例增强的设计选择，并为未来研究奠定了基础。