The remarkable advancements in large language models (LLMs) have significantly enhanced the performance in few-shot learning settings. By using only a small number of labeled examples, referred to as demonstrations, LLMs can effectively grasp the task at hand through in-context learning. However, the process of selecting appropriate demonstrations has received limited attention in prior work. This paper addresses the issue of identifying the most informative demonstrations for few-shot learning by approaching it as a pool-based Active Learning (AL) problem over a single iteration. Our objective is to investigate how AL algorithms can serve as effective demonstration selection methods for in-context learning. We compare various standard AL algorithms based on uncertainty, diversity, and similarity, and consistently observe that the latter outperforms all other methods, including random sampling. Notably, uncertainty sampling, despite its success in conventional supervised learning scenarios, performs poorly in this context. Our extensive experimentation involving a diverse range of GPT and OPT models across $24$ classification and multi-choice tasks, coupled with thorough analysis, unambiguously demonstrates that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.

翻译：大语言模型的显著进展极大地提升了少样本学习场景下的性能。通过仅使用少量标注样本（称为示范），大语言模型能够通过上下文学习有效掌握目标任务。然而，如何选择合适的示范在以往研究中受到较少关注。本文通过将少样本学习中信息量最大的示范识别问题建模为单轮次基于池的主动学习问题，系统研究了主动学习算法作为上下文学习示范选择方法的有效性。我们比较了基于不确定性、多样性和相似性的多种标准主动学习算法，并一致发现基于相似性的方法在所有对比方法中表现最优，包括随机采样。值得注意的是，在传统监督学习场景中取得成功的基于不确定性的采样在此背景下表现欠佳。通过在涵盖24个分类与多选任务的GPT和OPT模型系列上的广泛实验与深入分析，我们明确证明：基于主动学习的上下文示例选择能够优先筛选出兼具低不确定性与测试样本相似性的高质量示例。