In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. In this work, we adapt a recently proposed hardness metric, pointwise $\mathcal{V}$-usable information (PVI), to an in-context version (in-context PVI). Compared to the original PVI, in-context PVI is more efficient in that it requires only a few exemplars and does not require fine-tuning. We conducted a comprehensive empirical analysis to evaluate the reliability of in-context PVI. Our findings indicate that in-context PVI estimates exhibit similar characteristics to the original PVI. Specific to the in-context setting, we show that in-context PVI estimates remain consistent across different exemplar selections and numbers of shots. The variance of in-context PVI estimates across different exemplar selections is insignificant, which suggests that in-context PVI are stable. Furthermore, we demonstrate how in-context PVI can be employed to identify challenging instances. Our work highlights the potential of in-context PVI and provides new insights into the capabilities of ICL.

翻译：上下文学习（ICL）是一种伴随大语言模型发展而兴起的新型学习范式。在本工作中，我们将近期提出的困难度度量指标——逐点$\mathcal{V}$可用信息（PVI）——适配为上下文版本（上下文PVI）。相较于原始PVI，上下文PVI仅需少量示例且无需微调，因而更具效率。我们开展了全面的实证分析以评估上下文PVI的可靠性，结果表明其估计值与原始PVI具有相似特性。针对上下文学习的特殊场景，我们证实了上下文PVI估计值在不同示例选择与样本数量下保持一致性。不同示例选择所对应的上下文PVI估计值差异不显著，表明其具有稳定性。此外，我们展示了如何利用上下文PVI识别困难样本。本研究揭示了上下文PVI的应用潜力，并为理解ICL能力提供了新见解。