Despite the surprising few-shot performance of in-context learning (ICL), it is still a common practice to randomly sample examples to serve as context. This paper advocates a new principle for ICL: self-adaptive in-context learning. The self-adaption mechanism is introduced to help each sample find an in-context example permutation (i.e., selection and ordering) that can derive the correct prediction, thus maximizing performance. To validate the effectiveness of self-adaptive ICL, we propose a general select-then-rank framework and instantiate it with new selection and ranking algorithms. Upon extensive evaluation on eight different NLP datasets, our self-adaptive ICL method achieves a 40% relative improvement over the common practice setting. Further analysis reveals the enormous potential of self-adaptive ICL that it might be able to close the gap between ICL and finetuning given more advanced algorithms. Our code is released to facilitate future research in this area: https://github.com/Shark-NLP/self-adaptive-ICL

翻译：尽管上下文学习（ICL）在少样本场景下展现出令人惊讶的性能，但随机选取示例作为上下文仍是常见做法。本文提出ICL的新原则：自适应上下文学习。该自适应机制旨在帮助每个样本找到能够推导出正确预测的上下文示例排列（即选择与排序），从而最大化性能。为验证自适应ICL的有效性，我们提出通用"先选后排序"框架，并通过新型选择与排序算法实现该框架。在八个不同NLP数据集上的广泛评估表明，我们的自适应ICL方法相较于常见做法取得40%的相对性能提升。进一步分析揭示自适应ICL的巨大潜力：在更先进算法支持下，它或能弥合ICL与微调之间的性能差距。我们已公开代码以推动该领域后续研究：https://github.com/Shark-NLP/self-adaptive-ICL