In-context learning (ICL) ability has emerged with the increasing scale of large language models (LLMs), enabling them to learn input-label mappings from demonstrations and perform well on downstream tasks. However, under the standard ICL setting, LLMs may sometimes neglect query-related information in demonstrations, leading to incorrect predictions. To address this limitation, we propose a new paradigm called Hint-enhanced In-Context Learning (HICL) to explore the power of ICL in open-domain question answering, an important form in knowledge-intensive tasks. HICL leverages LLMs' reasoning ability to extract query-related knowledge from demonstrations, then concatenates the knowledge to prompt LLMs in a more explicit way. Furthermore, we track the source of this knowledge to identify specific examples, and introduce a Hint-related Example Retriever (HER) to select informative examples for enhanced demonstrations. We evaluate HICL with HER on 3 open-domain QA benchmarks, and observe average performance gains of 2.89 EM score and 2.52 F1 score on gpt-3.5-turbo, 7.62 EM score and 7.27 F1 score on LLaMA-2-Chat-7B compared with standard setting.

翻译：上下文学习能力随着大语言模型规模的扩大而涌现，使模型能够从示范中学习输入-标签映射关系，并在下游任务中表现良好。然而，在标准上下文学习设置下，大语言模型有时会忽略示范中与查询相关的信息，导致错误预测。为解决这一局限，我们提出名为提示增强上下文学习的新范式，以探索上下文学习在开放域问答（知识密集型任务的重要形式）中的潜力。提示增强上下文学习利用大语言模型的推理能力从示范中提取与查询相关的知识，然后将这些知识以更显式的方式拼接起来引导大语言模型。此外，我们追溯这些知识的来源以识别具体示例，并引入提示相关示例检索器来选取信息丰富的示例以增强示范。我们在3个开放域问答基准上评估了结合提示相关示例检索器的提示增强上下文学习方法，与标准设置相比，在gpt-3.5-turbo上平均获得2.89个EM分数和2.52个F1分数的提升，在LLaMA-2-Chat-7B上平均获得7.62个EM分数和7.27个F1分数的提升。