In-context learning is a new learning paradigm where a language model observes a few examples and then straightly outputs the test input's prediction. Previous works have shown that in-context learning is sensitive to the provided examples and randomly sampled examples show significantly unstable performance. In this paper, we propose to find ``supporting examples'' for in-context learning: Given the training dataset, we need to select one permutation of a few examples, which are informative for the task's in-context learning and lead to superior performance. Although in traditional gradient-based learning, e.g., fine-tuning, there are numerous methods to find a ``coreset'' from the entire dataset, they are sub-optimal and not suitable for this problem since in-context learning occurs in the language model's inference without gradients or parameter updates. Additionally, the strong dependence among in-context examples makes this problem an NP-hard combinatorial optimization problem and enumerating all possible permutations is infeasible. Hence we propose a two-stage method to tackle this challenge. First we propose a novel metric to select informative examples based on the language model's feedback, with a progressive filtering strategy. And then we propose a diversity-guided beam search method to refine and evaluate the selected examples, iteratively. The experimental results show our method significantly outperforms a wide range of baselines, and further analyses show the effectiveness of our method and shed light on the properties of supporting examples and in-context learning.

翻译：上下文学习是一种新的学习范式，其中语言模型观察少量示例后直接输出测试输入的预测。先前研究表明，上下文学习对所提供的示例敏感，随机采样的示例表现出显著不稳定的性能。本文提出为上下文学习寻找"支持示例"：给定训练数据集，我们需要选择一个由少量示例组成的排列，这些示例对任务的上下文学习具有信息性，并能带来优越的性能。尽管在传统基于梯度的学习（如微调）中，有众多方法可从整个数据集中寻找"核心集"，但这些方法在此问题中并非最优且不适用，因为上下文学习发生在语言模型的推理阶段，无需梯度或参数更新。此外，上下文示例之间的强依赖性使该问题成为NP难组合优化问题，枚举所有可能的排列不可行。因此，我们提出一种两阶段方法来解决这一挑战。首先，我们提出一种基于语言模型反馈的新型度量标准，结合渐进式过滤策略来选择信息性示例。然后，我们提出一种多样性引导的束搜索方法，迭代优化和评估所选示例。实验结果表明，我们的方法显著优于多种基线方法，进一步分析验证了该方法的有效性，并揭示了支持示例和上下文学习的特性。