Transformers have demonstrated remarkable success in natural language processing; however, their potential remains mostly unexplored for problems arising in dynamical systems. In this work, we investigate the optimal output estimation problem using transformers, which generate output predictions using all the past ones. We train the transformer using various systems drawn from a prior distribution and then evaluate its performance on previously unseen systems from the same distribution. As a result, the obtained transformer acts like a prediction algorithm that learns in-context and quickly adapts to and predicts well for different systems - thus we call it meta-output-predictor (MOP). MOP matches the performance of the optimal output estimator, based on Kalman filter, for most linear dynamical systems even though it does not have access to a model. We observe via extensive numerical experiments that MOP also performs well in challenging scenarios with non-i.i.d. noise, time-varying dynamics, and nonlinear dynamics like a quadrotor system with unknown parameters. To further support this observation, in the second part of the paper, we provide statistical guarantees on the performance of MOP and quantify the required amount of training to achieve a desired excess risk during test-time. Finally, we point out some limitations of MOP by identifying two classes of problems MOP fails to perform well, highlighting the need for caution when using transformers for control and estimation.

翻译：Transformer在自然语言处理领域取得了显著成功，然而其在动态系统相关问题上潜力仍基本未被探索。本文研究利用Transformer进行最优输出估计问题——该方法通过所有历史输出生成预测。我们使用源自先验分布的多种系统训练Transformer，随后评估其在同一分布中未见系统上的性能。由此获得的Transformer相当于一种能够上下文学习、快速适应不同系统并做出良好预测的算法——我们称之为元输出预测器（MOP）。即便不依赖系统模型，MOP对大多数线性动态系统的性能也能与基于卡尔曼滤波器的最优输出估计器相匹配。通过大量数值实验，我们观察到MOP在具有非独立同分布噪声、时变动态以及含未知参数的四旋翼飞行器等非线性动态系统的挑战性场景中同样表现优异。为支撑这一观察，论文第二部分给出了MOP性能的统计保证，并量化了达到测试期期望超额风险所需的训练量。最后，我们指出MOP在两类问题中表现欠佳，揭示了在控制与估计领域使用Transformer时需保持谨慎。