One emergent ability of large language models (LLMs) is that query-specific examples can be included in the prompt at inference time. In this work, we use active learning for adaptive prompt design and call it Active In-context Prompt Design (AIPD). We design the LLM prompt by adaptively choosing few-shot examples from a training set to optimize performance on a test set. The training examples are initially unlabeled and we obtain the label of the most informative ones, which maximally reduces uncertainty in the LLM prediction. We propose two algorithms, GO and SAL, which differ in how the few-shot examples are chosen. We analyze these algorithms in linear models: first GO and then use its equivalence with SAL. We experiment with many different tasks in small, medium-sized, and large language models; and show that GO and SAL outperform other methods for choosing few-shot examples in the LLM prompt at inference time.

翻译：大型语言模型（LLMs）的一项新兴能力是能够在推理时将查询特定示例纳入提示中。本研究利用主动学习进行自适应提示设计，并将其称为主动上下文提示设计（AIPD）。我们通过从训练集中自适应选择少样本示例来设计LLM提示，以优化测试集上的性能。训练示例最初未标注，我们通过获取信息量最大样本的标签来最大化降低LLM预测的不确定性。我们提出了GO和SAL两种算法，二者的区别在于少样本示例的选择策略。我们在线性模型中分析这些算法：首先分析GO算法，继而利用其与SAL算法的等价性展开研究。我们在小型、中型和大型语言模型中进行了多任务实验，结果表明在推理时选择LLM提示中的少样本示例时，GO和SAL算法均优于其他方法。