In recent years, attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. A key ingredient behind their success is the generative pretraining procedure, during which these models are trained on a large text corpus in an auto-regressive manner. To shed light on this phenomenon, we propose a new framework that allows both theory and systematic experiments to study the sequential modeling capabilities of transformers through the lens of Markov chains. Inspired by the Markovianity of natural languages, we model the data as a Markovian source and utilize this framework to systematically study the interplay between the data-distributional properties, the transformer architecture, the learnt distribution, and the final model performance. In particular, we theoretically characterize the loss landscape of single-layer transformers and show the existence of global minima and bad local minima contingent upon the specific data characteristics and the transformer architecture. Backed by experiments, we demonstrate that our theoretical findings are in congruence with the empirical results. We further investigate these findings in the broader context of higher order Markov chains and deeper architectures, and outline open problems in this arena. Code is available at \url{https://github.com/Bond1995/Markov}.

翻译：近年来，基于注意力的Transformer模型在自然语言等多个学科领域取得了巨大成功。其成功的关键因素之一是生成式预训练过程——该过程以自回归方式在大规模文本语料库上训练模型。为阐明这一现象，我们提出了一种新框架，该框架允许从理论和系统实验两个层面，通过马尔可夫链的视角研究Transformer的序列建模能力。受自然语言马尔可夫性质的启发，我们将数据建模为马尔可夫源，并利用该框架系统研究数据分布特性、Transformer架构、学习到的分布与最终模型性能之间的相互作用。具体而言，我们从理论上刻画了单层Transformer的损失景观，证明了在特定数据特征和Transformer架构条件下，全局极小值与不良局部极小值的存在性。通过实验验证，我们的理论发现与实证结果吻合。我们进一步将这些发现延伸至高阶马尔可夫链和更深层架构的广泛背景下，并指出该领域存在的开放性问题。代码已发布于\url{https://github.com/Bond1995/Markov}。