We study transformers' in-context learning of variable-length Markov chains (VOMCs), focusing on the finite-sample accuracy as the number of in-context examples increases. Compared to fixed-order Markov chains (FOMCs), learning VOMCs is substantially more challenging due to the additional structural learning component. The problem is naturally suited to a Bayesian formulation, where the context-tree weighting (CTW) algorithm, originally developed in the information theory community for universal data compression, provides an optimal solution. Empirically, we find that single-layer transformers fail to learn VOMCs in context, whereas transformers with two or more layers can succeed, with additional layers yielding modest but noticeable improvements. In contrast to prior results on FOMCs, attention-only networks appear insufficient for VOMCs. To explain these findings, we provide explicit transformer constructions: one with $D+2$ layers that can exactly implement CTW for VOMCs of maximum order $D$, and a simplified two-layer construction that uses partial information for approximate blending, shedding light on why two-layer transformers can perform well.

翻译：我们研究了Transformer对变阶马尔可夫链（VOMCs）的上下文学习能力，重点关注随上下文示例数量增加时的有限样本精度。与固定阶马尔可夫链（FOMCs）相比，由于VOMCs需要额外的结构学习组件，其学习难度显著提升。该问题自然适合采用贝叶斯框架，其中原本由信息论领域为通用数据压缩开发的上下文树加权（CTW）算法提供了最优解。实验发现，单层Transformer无法在上下文中学习VOMCs，而两层或更多层的Transformer可以成功，且额外层数能带来适度但显著的改进。与之前关于FOMCs的结果相反，纯注意力网络似乎不足以应对VOMCs。为解释这些发现，我们提供了显式Transformer构造：一种具有$D+2$层、可精确实现最大阶数$D$的VOMCs的CTW算法；另一种简化的两层构造则利用部分信息进行近似混合，揭示了两层Transformer为何能表现良好。