In-context learning is a new learning paradigm where a language model observes a few examples and then straightly outputs the test input's prediction. Previous works have shown that in-context learning is sensitive to the provided examples and randomly sampled examples show significantly unstable performance. In this paper, we propose to find ``supporting examples'' for in-context learning: Given the training dataset, we need to select one permutation of a few examples, which are informative for the task's in-context learning and lead to superior performance. Although in traditional gradient-based learning, e.g., fine-tuning, there are numerous methods to find a ``coreset'' from the entire dataset, they are sub-optimal and not suitable for this problem since in-context learning occurs in the language model's inference without gradients or parameter updates. Additionally, the strong dependence among in-context examples makes this problem an NP-hard combinatorial optimization problem and enumerating all possible permutations is infeasible. Hence we propose a two-stage method to tackle this challenge. First we propose a novel metric to select informative examples based on the language model's feedback, with a progressive filtering strategy. And then we propose a diversity-guided beam search method to refine and evaluate the selected examples, iteratively. The experimental results show our method significantly outperforms a wide range of baselines, and further analyses show the effectiveness of our method and shed light on the properties of supporting examples and in-context learning.

翻译：上下文学习是一种新的学习范式，其中语言模型观察少量示例后直接输出测试输入的预测。先前研究显示，上下文学习对所提供示例敏感，随机采样的示例会表现出显著不稳定的性能。本文提出为上下文学习寻找"支持示例"：给定训练数据集，我们需要选择少量示例的一种排列，这些示例对任务的上下文学习具有信息量并能带来优越性能。尽管在传统的基于梯度的学习（如微调）中，存在众多从整个数据集中寻找"核心集"的方法，但这些方法对当前问题而言是次优且不合适的，因为上下文学习发生在语言模型的推理过程中，不涉及梯度或参数更新。此外，上下文示例间的强依赖性使该问题成为NP难的组合优化问题，枚举所有可能排列是不可行的。因此我们提出两阶段方法应对这一挑战。首先，我们提出基于语言模型反馈的新型度量指标，结合渐进过滤策略来选择信息量丰富的示例。然后，我们提出多样性引导的束搜索方法迭代地精炼和评估所选示例。实验结果表明，我们的方法显著优于多种基线方法，进一步分析验证了方法的有效性，并揭示了支持示例的特性及上下文学习的本质。