To generate coherent responses, language models infer unobserved meaning from their input text sequence. One potential explanation for this capability arises from theories of delay embeddings in dynamical systems, which prove that unobserved variables can be recovered from the history of only a handful of observed variables. To test whether language models are effectively constructing delay embeddings, we measure the capacities of sequence models to reconstruct unobserved dynamics. We trained 1-layer transformer decoders and state-space sequence models on next-step prediction from noisy, partially-observed time series data. We found that each sequence layer can learn a viable embedding of the underlying system. However, state-space models have a stronger inductive bias than transformers-in particular, they more effectively reconstruct unobserved information at initialization, leading to more parameter-efficient models and lower error on dynamics tasks. Our work thus forges a novel connection between dynamical systems and deep learning sequence models via delay embedding theory.

翻译：为了生成连贯的响应，语言模型从其输入的文本序列中推断未观测到的语义。这种能力的一个潜在解释源于动力系统中的延迟嵌入理论，该理论证明仅从少数观测变量的历史中即可恢复未观测变量。为了检验语言模型是否有效地构建了延迟嵌入，我们测量了序列模型重建未观测动态的能力。我们使用带噪声、部分观测的时间序列数据进行下一步预测训练，训练对象包括单层Transformer解码器和状态空间序列模型。我们发现，每个序列层都能学习到对底层系统的一个可行嵌入。然而，状态空间模型比Transformer具有更强的归纳偏置——特别是，它们在初始化时能更有效地重建未观测信息，从而形成参数效率更高的模型，并在动态任务上获得更低的误差。因此，我们的工作通过延迟嵌入理论，在动力系统与深度学习序列模型之间建立了一种新颖的联系。