Positional encoding (PE) is widely viewed as necessary for transformers to process ordered sequences: without them, the next-token map appears permutation-invariant in its context tokens. This intuition underlies all prior universality results, which rely on positional information to prove that transformers with chain-of-thought can perform arbitrary computation, i.e., they are Turing complete. We revisit this belief in the regime most relevant to long-form reasoning, where generation proceeds through a finite sliding context window. Our opening perception is that the window mechanism itself (mildly) breaks the permutation symmetry. To distill and precisely capture the degree of this added expressiveness, we introduce an abstract autoregressive model, the HIST model, in which each update depends only on constant-size internal state and the token-count histogram within the current window. We prove that this HIST model is Turing complete by showing that the evolution of the window can reveal the token that has just left the window, which suffices to simulate Turing-complete Post machines. We then construct a sliding-window transformer over a constant-size token alphabet, without PE, and show that it can simulate the HIST model. Our result demonstrates that positional encodings are not indispensable for transformers to perform universal computation: The window sliding itself already breaks permutation symmetry and captures sufficient positional information.

翻译：位置编码（PE）被广泛视为Transformer处理有序序列的必要条件：若无位置编码，逐令牌映射在其上下文令牌中呈现置换不变性。这一直觉支撑了所有先前的通用性结论——这些结论依赖位置信息来证明具有思维链的Transformer能执行任意计算（即达到图灵完备性）。本文在与长程推理最相关的场景中重新审视这一信念：此时生成过程通过有限滑动上下文窗口进行。我们的初始观察发现，窗口机制本身（轻微地）打破了置换对称性。为了提炼并精确刻画这种新增表达能力的程度，我们引入了一种抽象自回归模型——HIST模型，其中每次更新仅依赖恒定大小的内部状态与当前窗口内的令牌计数直方图。我们通过证明窗口演化能揭示刚离开窗口的令牌（这足以模拟图灵完备的Post机），证得HIST模型具有图灵完备性。随后，我们在恒定大小令牌字母表上构建无PE的滑动窗口Transformer，并证明它能模拟HIST模型。我们的结果表明，Transformer实现通用计算并不必需位置编码：窗口滑动本身已能打破置换对称性并捕获充分的位序信息。