In-Context Learning (ICL) has significantly expanded the general-purpose nature of large language models, allowing them to adapt to novel tasks using merely the inputted context. This has motivated a series of papers that analyze tractable synthetic domains and postulate precise mechanisms that may underlie ICL. However, the use of relatively distinct setups that often lack a sequence modeling nature to them makes it unclear how general the reported insights from such studies are. Motivated by this, we propose a synthetic sequence modeling task that involves learning to simulate a finite mixture of Markov chains. As we show, models trained on this task reproduce most well-known results on ICL, hence offering a unified setting for studying the concept. Building on this setup, we demonstrate we can explain a model's behavior by decomposing it into four broad algorithms that combine a fuzzy retrieval vs. inference approach with either unigram or bigram statistics of the context. These algorithms engage in a competition dynamics to dominate model behavior, with the precise experimental conditions dictating which algorithm ends up superseding others: e.g., we find merely varying context size or amount of training yields (at times sharp) transitions between which algorithm dictates the model behavior, revealing a mechanism that explains the transient nature of ICL. In this sense, we argue ICL is best thought of as a mixture of different algorithms, each with its own peculiarities, instead of a monolithic capability. This also implies that making general claims about ICL that hold universally across all settings may be infeasible.

翻译：上下文学习（ICL）极大地扩展了大语言模型的通用性，使其能够仅利用输入的上下文来适应新任务。这促使了一系列论文通过分析可处理的合成领域，并假设可能支撑ICL的精确机制。然而，这些研究通常使用相对独立且往往缺乏序列建模本质的设置，使得所报告的见解的普适性尚不明确。受此启发，我们提出了一个合成序列建模任务，该任务涉及学习模拟一个有限混合的马尔可夫链。正如我们所示，在此任务上训练的模型复现了ICL大多数广为人知的结果，从而为研究这一概念提供了一个统一的框架。基于此设置，我们证明可以通过将模型行为分解为四种主要算法来解释其机制，这些算法结合了模糊检索与推理方法，并利用了上下文的单字或双字统计信息。这些算法通过竞争动力学来主导模型行为，具体的实验条件决定了哪种算法最终胜出：例如，我们发现仅仅改变上下文大小或训练量就会导致主导模型行为的算法之间发生（有时是急剧的）转变，这揭示了解释ICL瞬态性质的机制。从这个意义上说，我们认为ICL最好被视为多种不同算法的混合体，每种算法都有其独特之处，而非单一的整体能力。这也意味着，提出在所有设置中普遍成立的关于ICL的通用论断可能是不可行的。