In-context learning (ICL) suffers from oversensitivity to the prompt, which makes it unreliable in real-world scenarios. We study the sensitivity of ICL with respect to multiple types of perturbations. First, we find that label bias obscures true ICL sensitivity, and hence prior work may have significantly underestimated the true ICL sensitivity. Second, we observe a strong negative correlation between ICL sensitivity and accuracy, with sensitive predictions less likely to be correct. Motivated by these observations, we propose \textsc{SenSel}, a few-shot selective prediction method based on ICL sensitivity. Experiments on ten classification benchmarks show that \textsc{SenSel} consistently outperforms a commonly used confidence-based selective prediction baseline.

翻译：直截了当的学习(ICL)过于敏感,因此在现实世界的情景中不可靠。我们研究了ICL对多种扰动的敏感性。首先,我们发现标签上的偏差掩盖了ICL的真正敏感性,因此先前的工作可能大大低估了ICL的真正敏感性。第二,我们观察到ICL的敏感性和准确性之间有着强烈的负相关关系,而敏感预测不太可能正确。根据这些观察,我们提议采用基于ICL敏感性的几分镜头选择性预测方法,即“Textsc{SenSel} ” 。对十种分类基准的实验显示,“textsc{SenSel}” 一直比通常使用的基于信任的选择性预测基线要差。