Recent work has shown that small transformers trained in controlled "wind-tunnel'' settings can implement exact Bayesian inference, and that their training dynamics produce a geometric substrate -- low-dimensional value manifolds and progressively orthogonal keys -- that encodes posterior structure. We investigate whether this geometric signature persists in production-grade language models. Across Pythia, Phi-2, Llama-3, and Mistral families, we find that last-layer value representations organize along a single dominant axis whose position strongly correlates with predictive entropy, and that domain-restricted prompts collapse this structure into the same low-dimensional manifolds observed in synthetic settings. To probe the role of this geometry, we perform targeted interventions on the entropy-aligned axis of Pythia-410M during in-context learning. Removing or perturbing this axis selectively disrupts the local uncertainty geometry, whereas matched random-axis interventions leave it intact. However, these single-layer manipulations do not produce proportionally specific degradation in Bayesian-like behavior, indicating that the geometry is a privileged readout of uncertainty rather than a singular computational bottleneck. Taken together, our results show that modern language models preserve the geometric substrate that enables Bayesian inference in wind tunnels, and organize their approximate Bayesian updates along this substrate.

翻译：近期研究表明，在受控"风洞"环境中训练的小型Transformer能够实现精确的贝叶斯推断，其训练动态会产生几何基底——低维价值流形与渐进正交的键向量——用于编码后验结构。本研究探究这种几何特征是否存在于生产级语言模型中。通过对Pythia、Phi-2、Llama-3和Mistral系列模型的实验，我们发现末层价值表征沿单一主导轴组织，其位置与预测熵呈强相关性，且领域受限提示会将此结构坍缩为合成环境中观测到的相同低维流形。为探究该几何结构的功能，我们在Pythia-410M模型上进行上下文学习时对其熵对齐轴实施定向干预。移除或扰动该轴会选择性破坏局部不确定性几何结构，而匹配的随机轴干预则保持其完整。然而，这些单层操作并未产生相应比例的特异性贝叶斯行为退化，表明该几何结构是不确定性的特权读取机制，而非单一计算瓶颈。综合而言，我们的研究证明现代语言模型保留了风洞环境中实现贝叶斯推断的几何基底，并沿此基底组织其近似贝叶斯更新过程。