The remarkable advancements in large language models (LLMs) have significantly enhanced the performance in few-shot learning settings. By using only a small number of labeled examples, referred to as demonstrations, LLMs can effectively grasp the task at hand through in-context learning. However, the process of selecting appropriate demonstrations has received limited attention in prior work. This paper addresses the issue of identifying the most informative demonstrations for few-shot learning by approaching it as a pool-based Active Learning (AL) problem over a single iteration. Our objective is to investigate how AL algorithms can serve as effective demonstration selection methods for in-context learning. We compare various standard AL algorithms based on uncertainty, diversity, and similarity, and consistently observe that the latter outperforms all other methods, including random sampling. Notably, uncertainty sampling, despite its success in conventional supervised learning scenarios, performs poorly in this context. Our extensive experimentation involving a diverse range of GPT and OPT models across $24$ classification and multi-choice tasks, coupled with thorough analysis, unambiguously demonstrates that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.

翻译：大型语言模型（LLM）的最新进展显著提升了少样本学习场景下的性能。通过仅使用少量标注示例（即演示），LLM能够通过上下文学习有效掌握目标任务。然而，先前研究对如何选择合适的演示关注有限。本文通过将少样本学习中最具信息量的演示选择问题视为单轮次基于池的主动学习（AL）问题，对此进行了探讨。我们的目标是探究主动学习算法如何作为上下文学习的有效演示选择方法。我们比较了基于不确定性、多样性和相似性的多种标准主动学习算法，并一致发现后者表现优于所有其他方法（包括随机采样）。值得注意的是，不确定性采样虽然在传统监督学习场景中表现优异，但在本研究中效果不佳。我们在涵盖GPT和OPT系列模型的24个分类及多项选择任务上进行了广泛实验，并结合深入分析，明确证明：通过主动学习进行上下文示例选择时，优先选择了低不确定性且与测试示例具有相似性的高质量示例。