We reveal that transformers trained in an autoregressive manner naturally encode time-delayed causal structures in their learned representations. When predicting future values in multivariate time series, the gradient sensitivities of transformer outputs with respect to past inputs directly recover the underlying causal graph, without any explicit causal objectives or structural constraints. We prove this connection theoretically under standard identifiability conditions and develop a practical extraction method using aggregated gradient attributions. On challenging cases such as nonlinear dynamics, long-term dependencies, and non-stationary systems, this approach greatly surpasses the performance of state-of-the-art discovery algorithms, especially as data heterogeneity increases, exhibiting scaling potential where causal accuracy improves with data volume and heterogeneity, a property traditional methods lack. This unifying view lays the groundwork for a future paradigm where causal discovery operates through the lens of foundation models, and foundation models gain interpretability and enhancement through the lens of causality.

翻译：我们发现，以自回归方式训练的 Transformer 自然会在其学习到的表示中编码时间延迟的因果结构。在预测多元时间序列的未来值时，Transformer 输出相对于过去输入的梯度敏感性能够直接恢复底层的因果图，而无需任何显式的因果目标或结构约束。我们在标准的可识别性条件下从理论上证明了这种联系，并开发了一种使用聚合梯度归因的实用提取方法。在非线性动力学、长期依赖和非平稳系统等具有挑战性的案例中，该方法极大地超越了最先进的发现算法的性能，尤其是在数据异质性增加时，展现出因果准确性随数据量和异质性提升的扩展潜力，这是传统方法所不具备的特性。这种统一视角为未来的范式奠定了基础，即因果发现通过基础模型的视角进行，而基础模型则通过因果性的视角获得可解释性和增强。