Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.

翻译：语言模型，尤其是预训练的大语言模型，展现出作为少样本上下文学习者的卓越能力，能够仅凭借输入上下文中的少量示例便适应新任务。然而，模型执行上下文学习的能力对所选少样本示例的敏感性较高。最新进展之一是不再使用固定示例集，而是为每个输入查询检索定制化示例。检索示例的实现相对直接，可利用现有数据库和检索系统。这不仅提升了学习过程的效率和可扩展性，还被证明能减少人工选择示例时固有的偏差。鉴于检索式上下文学习取得的鼓舞性成果及不断增长的研究，我们对这一领域的研究开展了全面综述。本文讨论并比较了检索模型、检索训练过程及推理算法的不同设计选择。