Recently, large language models (LLMs) have made remarkable progress in natural language processing. The most representative ability of LLMs is in-context learning (ICL), which enables LLMs to learn patterns from in-context exemplars without training. The performance of ICL greatly depends on the exemplars used. However, how to choose exemplars remains unclear due to the lack of understanding of how in-context learning works. In this paper, we present a novel perspective on ICL by conceptualizing it as contextual retrieval from a model of associative memory. We establish a theoretical framework of ICL based on Hopfield Networks. Based on our framework, we look into how in-context exemplars influence the performance of ICL and propose more efficient active exemplar selection. Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval, with potential implications for advancing the understanding of LLMs.

翻译：近期，大型语言模型（LLMs）在自然语言处理领域取得了显著进展。LLMs最具代表性的能力是上下文学习（ICL），这使得LLMs能够在不进行训练的情况下，从上下文示例中学习模式。ICL的性能在很大程度上取决于所使用的示例。然而，由于对上下文学习工作机制的理解不足，如何选择示例仍不明确。本文通过将ICL概念化为从联想记忆模型中进行上下文检索，提出了一种新颖的视角。我们基于霍普菲尔德网络建立了ICL的理论框架。基于该框架，我们探究了上下文示例如何影响ICL的性能，并提出了更高效的主动示例选择方法。本研究通过将ICL与记忆检索联系起来，为理解其机制提供了新的启示，对深入认识LLMs具有潜在意义。