Recent work interprets the linear recoverability of geographic and temporal variables from large language model (LLM) hidden states as evidence for world-like internal representations. We test a simpler possibility: that much of the relevant structure is already latent in text itself. Applying the same class of ridge regression probes to static co-occurrence-based embeddings (GloVe and Word2Vec), we find substantial recoverable geographic signal and weaker but reliable temporal signal, with held-out R^2 values of 0.71-0.87 for city coordinates and 0.48-0.52 for historical birth years. Semantic-neighbor analyses and targeted subspace ablations show that these signals depend strongly on interpretable lexical gradients, especially country names and climate-related vocabulary. These findings suggest that ordinary word co-occurrence preserves richer spatial, temporal, and environmental structure than is often assumed, revealing a remarkable and underappreciated capacity of simple static embeddings to preserve world-shaped structure from text alone. Linear probe recoverability alone therefore does not establish a representational move beyond text.

翻译：近期研究将地理与时间变量从大语言模型（LLM）隐藏状态中的线性可恢复性解释为类世界内部表征的证据。我们检验了一种更简单的可能性：相关结构的大部分信息已潜在于文本自身。通过对基于静态共现的嵌入模型（GloVe 与 Word2Vec）应用同一类岭回归探针，我们发现存在大量可恢复的地理信号以及较弱但可靠的时序信号，其留一交叉验证的 R^2 值在城市坐标预测中达到 0.71–0.87，在历史人物出生年份预测中达到 0.48–0.52。语义邻近性分析与定向子空间消融实验表明，这些信号高度依赖于可解释的词汇梯度，尤其是国家名称和气候相关词汇。这些发现表明，普通词语共现关系保留了比通常假设更丰富的空间、时间及环境结构，揭示了简单静态嵌入模型仅从文本中保存世界形态结构的卓越且未被充分认识的能力。因此，仅凭线性探针的可恢复性并不能证明表征方式超越了文本本身。