In-context learning (ICL), teaching a large language model (LLM) to perform a task with few-shot demonstrations rather than adjusting the model parameters, has emerged as a strong paradigm for using LLMs. While early studies primarily used a fixed or random set of demonstrations for all test queries, recent research suggests that retrieving semantically similar demonstrations to the input from a pool of available demonstrations results in better performance. This work expands the applicability of retrieval-based ICL approaches by demonstrating that even simple word-overlap similarity measures such as BM25 outperform randomly selected demonstrations. Furthermore, we extend the success of retrieval-based ICL to instruction-finetuned LLMs as well as Chain-of-Thought (CoT) prompting. For instruction-finetuned LLMs, we find that although a model has already seen the training data at training time, retrieving demonstrations from the training data at test time yields better results compared to using no demonstrations or random demonstrations. Last but not least, we train a task-specific demonstration retriever that outperforms off-the-shelf retrievers.

翻译：摘要：上下文学习（ICL）是一种通过少量示例而非调整模型参数来教会大型语言模型（LLM）执行任务的范式，已成为利用LLM的重要方法。早期研究主要对所有测试查询使用固定或随机选取的示例集，而近期研究表明，从可用示例池中检索与输入语义相似的示例能够获得更优性能。本研究通过证明即使像BM25这样基于简单词重叠的相似度度量也能超越随机选取的示例，扩展了基于检索的ICL方法的适用范围。此外，我们将基于检索的ICL的成功应用延伸至指令微调的LLM以及思维链（CoT）提示方法。对于指令微调的LLM，我们发现尽管模型在训练时已接触过训练数据，但在测试时从训练数据中检索示例的效果仍优于使用无示例或随机示例的方法。最后但同样重要的是，我们训练了一个任务特定的示例检索器，其性能超越了现成的检索器。