Modeling spatiotemporal dynamical systems is a fundamental challenge in machine learning. Transformer models have been very successful in NLP and computer vision where they provide interpretable representations of data. However, a limitation of transformers in modeling continuous dynamical systems is that they are fundamentally discrete time and space models and thus have no guarantees regarding continuous sampling. To address this challenge, we present the Continuous Spatiotemporal Transformer (CST), a new transformer architecture that is designed for the modeling of continuous systems. This new framework guarantees a continuous and smooth output via optimization in Sobolev space. We benchmark CST against traditional transformers as well as other spatiotemporal dynamics modeling methods and achieve superior performance in a number of tasks on synthetic and real systems, including learning brain dynamics from calcium imaging data.

翻译：建模时空动态系统是机器学习中的一项基础性挑战。Transformer模型在自然语言处理和计算机视觉领域取得了巨大成功，能够为数据提供可解释的表征。然而，Transformer在建模连续动态系统时存在一个局限性：它们本质上是离散时间和空间模型，因此在连续采样方面无法提供保证。为解决这一挑战，我们提出了一种专门为连续系统建模而设计的新型Transformer架构——连续时空Transformer（CST）。该新框架通过在Sobolev空间中进行优化，确保了输出的连续性和平滑性。我们将CST与传统的Transformer及其他时空动态建模方法进行了基准测试，在合成数据和真实系统的多项任务中（包括从钙成像数据学习大脑动态）均取得了更优的性能。