In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) and revolutionized various fields by providing a few task-relevant demonstrations in the prompt. However, trustworthy issues with LLM's response, such as hallucination, have also been actively discussed. Existing works have been devoted to quantifying the uncertainty in LLM's response, but they often overlook the complex nature of LLMs and the uniqueness of in-context learning. In this work, we delve into the predictive uncertainty of LLMs associated with in-context learning, highlighting that such uncertainties may stem from both the provided demonstrations (aleatoric uncertainty) and ambiguities tied to the model's configurations (epistemic uncertainty). We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. Extensive experiments are conducted to demonstrate the effectiveness of the decomposition. The code and data are available at: \url{https://github.com/lingchen0331/UQ_ICL}.

翻译：上下文学习已成为大语言模型的一项突破性能力，通过在提示中提供少量任务相关示例，这一能力推动了多个领域的革新。然而，大语言模型响应中的可信度问题（如幻觉现象）也引发了广泛讨论。现有研究致力于量化大语言模型响应中的不确定性，但往往忽视了大语言模型的复杂本质以及上下文学习的独特性。本文深入探究与大语言模型上下文学习相关的预测不确定性，强调此类不确定性可能源自所提供的示例（随机不确定性）以及模型配置相关的模糊性（认知不确定性）。我们提出一种新型公式化表达及相应的估计方法，以量化这两类不确定性。所提方法提供了一种无监督方式，以即插即用的形式理解上下文学习的预测结果。通过大量实验验证了解耦方法的有效性。代码与数据获取链接：\url{https://github.com/lingchen0331/UQ_ICL}。