In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompts across different models and tasks. To address this gap, we present a comprehensive prompt analysis based on the sensitivity of a function. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL.

翻译：上下文学习（ICL）已成为最流行的学习范式之一。尽管关于提示工程的文献日益增多，但缺乏系统性的分析来比较不同模型和任务下提示的效果。为填补这一空白，我们基于函数的敏感性提出了一种全面的提示分析。我们的分析表明，敏感性是模型性能的无监督代理指标，因为其与准确性呈强负相关。我们利用基于梯度的显著性分数，实证展示了不同提示如何影响输入标记与输出的相关性，从而产生不同水平的敏感性。此外，我们引入了敏感性感知解码，将敏感性估计作为惩罚项纳入标准贪心解码中。我们证明，当输入信息稀缺时，这种方法尤其有效。我们的工作为提示分析提供了全新视角，并有助于更深入地理解ICL的机制。