In-context learning with large language models (LLMs) excels at adapting to various tasks rapidly. However, its success hinges on carefully selecting demonstrations, which remains an obstacle in practice. Current approaches to this problem either rely on hard-to-acquire external supervision or require frequent interactions with LLMs, resulting in high costs. We propose a new method called In-Context Reflection (ICR) to overcome these challenges. ICR strategically selects demonstrations to reduce the discrepancy between the LLM's outputs and the actual input-output mappings. Specifically, ICR starts with a random set of initial demonstrations, then iteratively refines it. In each step, it analyzes a pool of candidate examples and identifies the ones most likely to challenge the LLM's current understanding, measured by a new metric called misconfidence. These most confusing examples are then selected to replace the less informative demonstrations in the current set. Our comprehensive evaluation across five diverse datasets encompassing 13 subtasks shows the efficacy of ICR. Compared to existing methods, ICR achieves an average performance boost of 4%, while demonstrating remarkable cross-task generalization capabilities.

翻译：基于大语言模型（LLMs）的上下文学习在快速适应各类任务方面表现卓越。然而，其效果高度依赖对示例进行精心筛选，而这在实践中仍是一大障碍。现有方法或依赖难以获取的外部监督，或需与LLMs频繁交互导致成本高昂。为克服上述挑战，我们提出名为"上下文反思"（In-Context Reflection，ICR）的新方法。ICR通过策略性地选择示例，降低LLM输出与实际输入-输出映射之间的偏差。具体而言，ICR首先使用随机初始示例集，随后迭代优化该集合。每次迭代中，方法分析候选示例池，通过新提出的"误信度"（misconfidence）指标识别最有可能挑战LLM当前理解的示例。这些最具迷惑性的示例被选入当前集合，替换信息量较低的示例。我们在涵盖13个子任务的五个多样化数据集上的综合评估验证了ICR的有效性。与现有方法相比，ICR在实现平均性能提升4%的同时，展现出卓越的跨任务泛化能力。