In-context learning (ICL) i.e. showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL. One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task. From prior work (Min et al., 2022), we already know that labels paired with the demonstrations bias the model predictions. This leads us to our hypothesis whether considering LLM's existing knowledge about the task, especially with respect to the output label space can help in a better demonstration selection strategy. Through extensive experimentation on three text classification tasks, we find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those demonstrations that help resolve the inherent label ambiguity surrounding the test example. Interestingly, we find that including demonstrations that the LLM previously mis-classified and also fall on the test example's decision boundary, brings the most performance gain.

翻译：语境学习（In-context learning, ICL）即仅向大型语言模型（LLM）提供少量任务特定示例，无需任务特定微调即可提升下游任务表现。然而，LLM对提示的选择具有敏感性，因此关键研究问题在于如何为ICL选择优质示例。一种有效策略是利用文本检索器建立ICL示例与测试输入之间的语义相似性，但该方法未考虑LLM对该任务的已有知识，因而存在次优性。基于先前研究（Min等人，2022），我们已知与示例配对的标签会偏置模型预测。据此提出假设：考虑LLM对任务的已有知识（特别是输出标签空间方面的知识）能否优化示例选择策略。通过在三项文本分类任务上的广泛实验，我们发现不仅应选择语义相似的ICL示例，更应选取有助于消解测试样本固有标签歧义的示例。有趣的是，若纳入LLM先前分类错误且位于测试样本决策边界上的示例，可获得最大性能提升。