In-Context Learning (ICL) and Instruction Tuning (IT) are two primary paradigms of adopting Large Language Models (LLMs) to downstream applications. However, they are significantly different. In ICL, a set of demonstrations are provided at inference time but the LLM's parameters are not updated. In IT, a set of demonstrations are used to tune LLM's parameters in training time but no demonstrations are used at inference time. Although a growing body of literature has explored ICL and IT, studies on these topics have largely been conducted in isolation, leading to a disconnect between these two paradigms. In this work, we explore the relationship between ICL and IT by examining how the hidden states of LLMs change in these two paradigms. Through carefully designed experiments conducted with LLaMA-2 (7B and 13B), we find that ICL is implicit IT. In other words, ICL changes an LLM's hidden states as if the demonstrations were used to instructionally tune the model. Furthermore, the convergence between ICL and IT is largely contingent upon several factors related to the provided demonstrations. Overall, this work offers a unique perspective to explore the connection between ICL and IT and sheds light on understanding the behaviors of LLM.

翻译：上下文学习（ICL）和指令微调（IT）是将大语言模型（LLMs）应用于下游任务的两种主要范式。然而，它们之间存在显著差异。在ICL中，推理时提供一组示例，但LLM的参数不会更新；而在IT中，训练时使用一组示例来调整LLM的参数，但推理时不使用任何示例。尽管已有越来越多的文献探讨ICL和IT，但这些研究大多相互独立进行，导致这两种范式之间存在脱节。在本工作中，我们通过研究LLM隐藏状态在这两种范式中的变化，探索ICL与IT之间的关系。通过使用LLaMA-2（7B和13B）进行的精心设计实验，我们发现ICL本质上是隐式的IT。换言之，ICL对LLM隐藏状态的改变，仿佛这些示例被用于对模型进行指令微调。此外，ICL与IT之间的趋同程度很大程度上取决于所提供示例的多个相关因素。总体而言，本工作为探索ICL与IT之间的联系提供了独特视角，并有助于理解LLM的行为。