Exchangeability-based martingale diagnostics have been used to question Bayesian explanations of transformer in-context learning. We show that these violations are compatible with Bayesian/MDL behavior once we account for a basic architectural fact: positional encodings break exchangeability. Accordingly, the relevant baseline is performance in expectation over orderings of an exchangeable multiset, not performance under every fixed ordering. In a Bernoulli microscope (under explicit regularity assumptions), we bound the permutation-induced dispersion detected by martingale diagnostics (Theorem~3.4) while proving near-optimal expected MDL/compression over permutations (Theorem~3.6). Empirically, black-box next-token log-probabilities from an Azure OpenAI deployment exhibit nonzero expectation--realization gaps that decay with context length (mean 0.74 at $n = 10$ to 0.26 at $n = 50$; 95\% confidence intervals), and permutation averaging reduces order-induced standard deviation with a $k^{-1/2}$ trend (Figure~2). Controlled from-scratch training ablations varying only the positional encoding show within-prefix order variance collapsing to $\approx 10^{-16}$ with no positional encoding, but remaining $10^{-8}$--$10^{-6}$ under standard positional encoding schemes (Table~2). Robustness checks extend beyond Bernoulli to categorical sequences, synthetic in-context learning tasks, and evidence-grounded QA with permuted exchangeable evidence chunks.

翻译：基于可交换性的鞅诊断方法曾被用于质疑Transformer上下文学习的贝叶斯解释。我们证明，一旦考虑一个基本架构事实——位置编码会破坏可交换性，这些违反现象与贝叶斯/最小描述长度（MDL）行为是相容的。因此，相关的基准应是在可交换多重集的顺序排列上的期望性能，而非每个固定顺序下的性能。在伯努利显微镜模型（基于明确的正则性假设）中，我们界定了鞅诊断检测到的置换诱导离散度（定理3.4），同时证明了在置换上接近最优的期望MDL/压缩性能（定理3.6）。实证研究表明，Azure OpenAI部署的黑盒下一词元对数概率表现出非零的期望-实现差距，且该差距随上下文长度衰减（均值从$n=10$时的0.74降至$n=50$时的0.26；95%置信区间），而置换平均能以$k^{-1/2}$的趋势降低顺序诱导的标准差（图2）。通过仅改变位置编码的受控从头训练消融实验显示：无位置编码时前缀内顺序方差坍缩至约$10^{-16}$，而在标准位置编码方案下仍保持在$10^{-8}$—$10^{-6}$之间（表2）。鲁棒性检验从伯努利序列扩展到分类序列、合成上下文学习任务，以及具有可置换可交换证据块的证据基础问答。