The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a coherent model of the data generation process -- a world model. We find preliminary evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual ``space neurons'' and ``time neurons'' that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.

翻译：大型语言模型（LLM）的能力引发了关于这类系统是否仅学习了海量表面统计数据，还是形成了对数据生成过程（即世界模型）的连贯表征的争论。通过分析Llama-2系列模型中三类空间数据集（全球、美国、纽约市地点）与三类时间数据集（历史人物、艺术品、新闻标题）的学习表征，我们发现了支持后者的初步证据。研究表明，LLM在多个尺度上学习了空间与时间的线性表征。这些表征对提示变化具有鲁棒性，并在不同实体类型（如城市与地标）间保持统一。此外，我们识别出独立的"空间神经元"与"时间神经元"，它们能够可靠地编码空间与时间坐标。尽管仍需进一步研究，我们的结果表明现代LLM学习了真实世界丰富的时空表征，并具备了世界模型的基本要素。