Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. In this paper, we study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

翻译：预训练的大型语言模型（LLMs）在执行零样本任务（包括时间序列预测）方面表现出惊人的有效性。然而，由于这些模型的复杂性，理解其背后的机制仍然极具挑战性。本文研究了LLMs推断受物理兴趣原理支配演化的动力系统行为的能力。我们的结果表明，主要基于文本训练的语言模型LLaMA 2，在无需微调或提示工程的情况下，能够准确预测动力系统的时间序列。此外，所学物理规则的准确性随输入上下文窗口长度的增加而提高，揭示了上下文中的神经缩放定律。在此过程中，我们提出了一种灵活高效的算法，可直接从LLMs中提取多位数字的概率密度函数。