The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a coherent model of the data generating process -- a world model. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual ``space neurons'' and ``time neurons'' that reliably encode spatial and temporal coordinates. Our analysis demonstrates that modern LLMs acquire structured knowledge about fundamental dimensions such as space and time, supporting the view that they learn not merely superficial statistics, but literal world models.

翻译：大语言模型（LLMs）的能力引发了一场争论：这些系统仅仅是学习了海量的表面统计数据，还是学习了数据生成过程的内在模型——即世界模型。通过分析Llama-2系列模型在三个空间数据集（世界、美国、纽约市地点）和三个时间数据集（历史人物、艺术品、新闻标题）上的学习表征，我们发现了支持后者的证据。我们发现，LLMs在多个尺度上学习了空间和时间的线性表征。这些表征对于提示词的变化具有鲁棒性，并且在不同实体类型（如城市和地标）之间是统一的。此外，我们还识别出能够可靠编码空间和时间坐标的单个“空间神经元”和“时间神经元”。我们的分析表明，现代LLMs掌握了关于空间和时间等基本维度的结构化知识，这支持了它们不仅学习表面统计数据，而是学习字面世界模型的观点。