In-Context Learning (ICL) over Large language models (LLMs) aims at solving previously unseen tasks by conditioning on a few training examples, eliminating the need for parameter updates and achieving competitive performance. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets, i.e., the inherent knowledge learned in LLMs, the factual knowledge derived from the selected in-context examples, and the knowledge biases in LLMs for output generation. To unleash the power of LLMs in few-shot learning scenarios, we introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL: 1) injecting factual knowledge to LLMs during continual self-supervised pre-training, 2) judiciously selecting the examples with high knowledge relevance, and 3) calibrating the prediction results based on prior knowledge. We evaluate the proposed approaches on auto-regressive LLMs (e.g., GPT-style models) over multiple text classification and question answering tasks. Experimental results demonstrate that KICT substantially outperforms strong baselines, and improves by more than 13% and 7% of accuracy on text classification and question answering tasks, respectively.

翻译：上下文学习（ICL）旨在通过依赖少量训练示例使大语言模型（LLMs）适应先前未见过的任务，从而无需参数更新即可实现具有竞争力的性能。本文从三个核心维度论证了事实知识对ICL性能的关键作用，即：LLMs中固有的知识、从所选上下文示例中提取的事实知识，以及LLMs在输出生成中存在的知识偏差。为了在少样本学习场景中充分发挥大语言模型的潜力，我们提出了一种新颖的知识感知上下文调优（KICT）框架，通过以下方式进一步提升ICL性能：1）在持续自监督预训练阶段向LLMs注入事实知识，2）审慎选择知识相关性高的示例，3）基于先验知识校准预测结果。我们在自回归大语言模型（如GPT类模型）上，针对多项文本分类和问答任务评估了所提方法。实验结果表明，KICT显著优于强基线模型，在文本分类和问答任务上的准确率分别提升超过13%和7%。