Large language models are able to learn new tasks in context, where they are provided with instructions and a few annotated examples. However, the effectiveness of in-context learning is dependent on the provided context, and the performance on a downstream task can vary considerably, depending on the instruction. Importantly, such dependency on the context can surface in unpredictable ways, e.g., a seemingly more informative instruction might lead to a worse performance. In this paper, we propose an alternative approach, which we term In-Context Probing (ICP). Similar to in-context learning, we contextualize the representation of the input with an instruction, but instead of decoding the output prediction, we probe the contextualized representation to predict the label. Through a series of experiments on a diverse set of classification tasks, we show that in-context probing is significantly more robust to changes in instructions. We further show that ICP performs competitive or superior to finetuning and can be particularly helpful to build classifiers on top of smaller models, with less than a hundred training examples.

翻译：大型语言模型能够通过上下文学习新任务，即通过提供指令和少量标注示例实现。然而，上下文学习的有效性依赖于所提供的上下文，且下游任务的性能可能因指令不同而显著变化。重要的是，这种对上下文的依赖性可能以不可预测的方式显现，例如，看似信息更丰富的指令反而可能导致性能下降。本文提出一种替代方法，称为上下文探测（In-Context Probing, ICP）。与上下文学习类似，我们通过指令对输入表示进行上下文化，但不同于解码输出预测，我们探测上下文化表示以预测标签。通过在多种分类任务上的一系列实验，我们证明上下文探测对指令变化具有显著更强的鲁棒性。进一步研究表明，ICP在性能上与微调相当或更优，尤其有助于在少于百个训练样本的情况下，基于较小模型构建分类器。