We investigate whether in-context examples, widely used in decoder-only language models (LLMs), can improve embedding model performance in retrieval tasks. Unlike in LLMs, naively prepending in-context examples (query-document pairs) to the target query at inference time does not work out of the box. We introduce a simple approach to enable retrievers to use in-context examples. Our approach, RARe, finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This can be applied to adapt various base architectures (i.e., decoder-only language models, retriever models) and consistently achieves performance gains of up to +2.72% nDCG across various open-domain retrieval datasets (BeIR, RAR-b). In particular, we find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. We further provide analysis on the design choices of in-context example augmentation and lay the foundation for future work in this space.

翻译：我们研究了在仅解码器语言模型（LLMs）中广泛使用的上下文示例是否能够提升嵌入模型在检索任务中的性能。与LLMs不同，在推理阶段简单地将上下文示例（查询-文档对）直接前置到目标查询中并不能直接生效。我们提出了一种简单方法，使检索模型能够利用上下文示例。我们的方法RARe通过微调预训练模型实现，所使用的上下文示例在语义上与目标查询相似。该方法可适配多种基础架构（如仅解码器语言模型、检索器模型），并在多个开放域检索数据集（BeIR、RAR-b）上持续实现高达+2.72% nDCG的性能提升。特别值得注意的是，与使用无上下文示例查询的模型相比，RARe展现出更强的跨领域泛化能力，这与LLMs中上下文学习的表现相似。我们进一步分析了上下文示例增强的设计选择，并为该领域的后续研究奠定了基础。