In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) and revolutionized various fields by providing a few task-relevant demonstrations in the prompt. However, trustworthy issues with LLM's response, such as hallucination, have also been actively discussed. Existing works have been devoted to quantifying the uncertainty in LLM's response, but they often overlook the complex nature of LLMs and the uniqueness of in-context learning. In this work, we delve into the predictive uncertainty of LLMs associated with in-context learning, highlighting that such uncertainties may stem from both the provided demonstrations (aleatoric uncertainty) and ambiguities tied to the model's configurations (epistemic uncertainty). We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. Extensive experiments are conducted to demonstrate the effectiveness of the decomposition. The code and data are available at: https://github.com/lingchen0331/UQ_ICL.

翻译：上下文学习已成为大型语言模型的一项开创性能力，通过在提示中提供少量任务相关的示例，彻底改变了多个领域。然而，大型语言模型响应的可信度问题，如幻觉现象，也引发了广泛讨论。现有工作致力于量化大型语言模型响应的不确定性，但往往忽略了大型语言模型的复杂本质以及上下文学习的独特性。在本工作中，我们深入探究大型语言模型在上下文学习中涉及的预测不确定性，强调此类不确定性可能源自提供的示例（偶然不确定性）以及与模型配置相关的歧义（认知不确定性）。我们提出了一种新的形式化表述及相应的估计方法来量化这两种类型的不确定性。所提出的方法提供了一种无监督的方式，以即插即用的形式理解上下文学习的预测。我们通过大量实验验证了该分解方法的有效性。代码和数据已开源：https://github.com/lingchen0331/UQ_ICL。