The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.

翻译：GPT-3等大型语言模型的上下文学习能力使标注者能够通过少量示例为特定任务定制大型语言模型。然而，用户在构建示例时往往只包含最明显的模式，导致上下文函数定义不充分，难以应对未见案例。此外，即使针对已知模式，也很难判断何时已包含"足够"的示例。本研究中，我们提出ScatterShot——一种为上下文学习构建高质量演示集的交互式系统。ScatterShot通过迭代将未标注数据划分为任务特定模式，以主动学习方式从尚未充分探索或未饱和的数据切片中采样信息丰富的输入，并借助大型语言模型和当前示例集帮助用户更高效地进行标注。在两个文本扰动场景的仿真研究中，相较于随机采样，ScatterShot采样方法使所得的小样本函数性能提升4-5个百分点，且随着示例增加方差更小。用户研究表明，ScatterShot显著帮助用户覆盖输入空间中的不同模式并更高效地标注上下文示例，从而提升上下文学习效果并减少用户工作量。