Understanding in-context learning (ICL) capability that enables large language models (LLMs) to excel in proficiency through demonstration examples is of utmost importance. This importance stems not only from the better utilization of this capability across various tasks, but also from the proactive identification and mitigation of potential risks, including concerns regarding truthfulness, bias, and toxicity, that may arise alongside the capability. In this paper, we present a thorough survey on the interpretation and analysis of in-context learning. First, we provide a concise introduction to the background and definition of in-context learning. Then, we give an overview of advancements from two perspectives: 1) a theoretical perspective, emphasizing studies on mechanistic interpretability and delving into the mathematical foundations behind ICL; and 2) an empirical perspective, concerning studies that empirically analyze factors associated with ICL. We conclude by highlighting the challenges encountered and suggesting potential avenues for future research. We believe that our work establishes the basis for further exploration into the interpretation of in-context learning. Additionally, we have created a repository containing the resources referenced in our survey.

翻译：理解上下文学习（ICL）能力——即通过演示示例使大型语言模型（LLMs）在熟练度方面表现出色的能力——至关重要。这种重要性不仅源于在各种任务中更好地利用该能力，还源于主动识别和缓解伴随该能力可能出现的潜在风险，包括有关真实性、偏见和毒性的担忧。在本文中，我们对上下文学习的解释与分析进行了全面综述。首先，我们简要介绍了上下文学习的背景和定义。然后，我们从两个角度概述了研究进展：1）理论视角，侧重于机制可解释性研究，并深入探讨ICL背后的数学基础；2）实证视角，涉及对与ICL相关因素进行实证分析的研究。最后，我们强调了当前面临的挑战，并提出了未来研究的潜在方向。我们相信，我们的工作为进一步探索上下文学习的解释奠定了基础。此外，我们还创建了一个包含本综述所引用资源的存储库。