Large language models are able to learn new tasks in context, where they are provided with instructions and a few annotated examples. However, the effectiveness of in-context learning is dependent on the provided context, and the performance on a downstream task can vary considerably, depending on the instruction. Importantly, such dependency on the context can surface in unpredictable ways, e.g., a seemingly more informative instruction might lead to a worse performance. In this paper, we propose an alternative approach, which we term in-context probing. Similar to in-context learning, we contextualize the representation of the input with an instruction, but instead of decoding the output prediction, we probe the contextualized representation to predict the label. Through a series of experiments on a diverse set of classification tasks, we show that in-context probing is significantly more robust to changes in instructions. We further show that probing performs competitive or superior to finetuning and can be particularly helpful to build classifiers on top of smaller models, and with only a hundred training examples.

翻译：大型语言模型能够在上下文中学习新任务，即通过提供的指令和少量标注示例进行学习。然而，上下文学习的效果依赖于所提供的上下文，且下游任务的性能可能因指令不同而显著变化。重要的是，这种对上下文的依赖性可能以不可预测的方式显现，例如，看似更具信息量的指令反而可能导致性能下降。本文提出一种替代方法，称为“上下文探测”。与上下文学习类似，我们通过指令对输入表示进行上下文化处理，但并非解码输出预测，而是探测上下文化表示以预测标签。通过在多种分类任务上的一系列实验，我们表明上下文探测对指令变化具有显著更高的鲁棒性。我们进一步证明，探测方法的表现与微调相当或更优，尤其在仅使用少量（例如一百个）训练样本且基于较小模型构建分类器时尤为有效。