Match-and-copy is a core retrieval primitive used at inference time by large language models to retrieve a matching token from the context then copy its successor. Yet, understanding how this behavior emerges on natural data is challenging because retrieval and memorization are entangled. To disentangle the two, we introduce Gaussian Match-and-Copy (GMC), a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals. Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits in practice, and separates architectures by their retrieval capabilities. We also analyze the optimization dynamics in a simplified attention setting. Although many solutions are a priori possible under a regression objective, including ones that do not implement retrieval, we identify an implicit-bias regime in which gradient descent drives the parameters to diverge while their direction aligns with the max-margin separator, yielding hard match selection. We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.

翻译：匹配复制是大型语言模型在推理时使用的核心检索原语，用于从上下文中检索匹配的标记并复制其后继标记。然而，理解这种行为在自然数据上如何形成具有挑战性，因为检索与记忆机制相互交织。为区分二者，我们提出了高斯匹配复制基准——一个通过纯二阶相关信号隔离长程检索的最小化基准。数值研究表明，该任务保留了Transformer在实践中形成匹配复制电路的关键质性特征，并能根据架构的检索能力进行区分。我们还在简化注意力设置中分析了优化动态。尽管回归目标在理论上允许多种解决方案（包括不实现检索的方案），我们识别出一个隐式偏置机制：在该机制下，梯度下降会驱动参数发散，同时其方向与最大间隔分类器对齐，从而产生硬匹配选择。我们针对在明确技术条件下达到经验损失消失的梯度下降轨迹，证明了这种最大间隔对齐性质。