Large language models show a surprising in-context learning ability -- being able to use a prompt to form a prediction for a query, yet without additional training, in stark contrast to old-fashioned supervised learning. Providing a mechanistic interpretation and linking the empirical phenomenon to physics are thus challenging and remain unsolved. We study a simple yet expressive transformer with linear attention and map this structure to a spin glass model with real-valued spins, where the couplings and fields explain the intrinsic disorder in data. The spin glass model explains how the weight parameters interact with each other during pre-training, and further clarifies why an unseen function can be predicted by providing only a prompt yet without further training. Our theory reveals that for single-instance learning, increasing the task diversity leads to the emergence of in-context learning, by allowing the Boltzmann distribution to converge to a unique correct solution of weight parameters. Therefore the pre-trained transformer displays a prediction power in a novel prompt setting. The proposed analytically tractable model thus offers a promising avenue for thinking about how to interpret many intriguing but puzzling properties of large language models.

翻译：大型语言模型展现出令人惊讶的上下文学习能力——能够利用提示对查询形成预测，而无需额外训练，这与传统的监督学习形成鲜明对比。为此现象提供机制性解释并将其与物理学原理联系起来，是一项具有挑战性且尚未解决的课题。我们研究了一个结构简洁但表达能力强的线性注意力Transformer，并将该结构映射到一个具有实值自旋的自旋玻璃模型，其中耦合项和场项解释了数据的内在无序性。该自旋玻璃模型阐释了权重参数在预训练过程中如何相互作用，并进一步阐明了为何仅通过提供提示（无需额外训练）即可预测未见过的函数。我们的理论表明，在单实例学习场景中，增加任务多样性会促使玻尔兹曼分布收敛至权重参数的唯一正确解，从而催生上下文学习能力的涌现。因此，预训练的Transformer能够在全新的提示设置下展现预测能力。这一可解析处理的模型为理解大型语言模型诸多引人入胜却又令人困惑的特性，提供了一条前景广阔的研究路径。