Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

翻译：预训练大语言模型（LLMs）在执行零样本任务（包括时间序列预测）方面展现出惊人的有效性。然而，由于模型复杂性，理解这种能力背后的机制仍极具挑战性。我们研究LLMs外推受物理学原理控制的动力系统行为的能力。结果表明，主要基于文本训练的LLaMA 2语言模型无需微调或提示工程即可实现动力系统时间序列的准确预测。此外，所学物理规则的准确性随输入上下文窗口长度的增加而提升，这揭示了一种上下文版本的神经缩放定律。在此过程中，我们提出了一种灵活高效的算法，可直接从LLMs中提取多位数数字的概率密度函数。