Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by a few input-output examples. However, prior research has shown that in-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. Therefore, the construction of an appropriate prompt is essential for improving the performance of in-context learning. In this paper, we revisit this problem from the view of predictive bias. Specifically, we introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's in-context learning performance in an effective and interpretable manner.

翻译：大语言模型已展现出进行上下文学习的惊人能力，即通过基于少量输入-输出示例构建的提示，这些模型可直接应用于解决众多下游任务。然而，先前研究表明，由于训练示例、示例顺序和提示格式的变化，上下文学习可能面临高度不稳定性。因此，构建合适的提示对于提升上下文学习性能至关重要。本文从预测偏差的角度重新审视这一问题。具体而言，我们引入了一个度量标准，用于评估固定提示相对于标签或给定属性的预测偏差。然后，我们通过实验证明，具有较高偏差的提示往往导致不佳的预测质量。基于这一发现，我们提出了一种基于贪婪搜索的新型搜索策略，以识别近似最优的提示，从而提升上下文学习性能。我们使用GPT-3等最先进的主流模型，在多种下游任务上进行了全面实验。结果表明，我们的方法能够以有效且可解释的方式增强模型的上下文学习性能。