In-context learning, a capability that enables a model to learn from input examples on the fly without necessitating weight updates, is a defining characteristic of large language models. In this work, we follow the setting proposed in (Garg et al., 2022) to better understand the generality and limitations of in-context learning from the lens of the simple yet fundamental task of linear regression. The key question we aim to address is: Are transformers more adept than some natural and simpler architectures at performing in-context learning under varying distribution shifts? To compare transformers, we propose to use a simple architecture based on set-based Multi-Layer Perceptrons (MLPs). We find that both transformers and set-based MLPs exhibit in-context learning under in-distribution evaluations, but transformers more closely emulate the performance of ordinary least squares (OLS). Transformers also display better resilience to mild distribution shifts, where set-based MLPs falter. However, under severe distribution shifts, both models' in-context learning abilities diminish.

翻译：上下文学习是一种使模型能够在无需权重更新的情况下即时从输入示例中学习的能力，这是大型语言模型的一个定义性特征。在本工作中，我们沿用（Garg等人，2022）提出的设置，通过简单而基础的线性回归任务视角，以更好地理解上下文学习的通用性与局限性。我们旨在解决的关键问题是：在各类分布漂移下，Transformer是否比某些自然且更简单的架构更擅长执行上下文学习？为进行对比，我们提出使用基于集合的多层感知机（MLPs）这一简单架构。研究发现，在分布内评估中，Transformer和基于集合的MLP均表现出上下文学习能力，但Transformer的性能更接近普通最小二乘法（OLS）。Transformer在轻度分布漂移下展现出更强的鲁棒性，而基于集合的MLP在此情况下表现不佳。然而，在严重分布漂移下，两种模型的上下文学习能力均会减弱。