Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

翻译：建模不规则时间序列上的连续时间动态，对于解释数据随时间持续演化的规律及关联性至关重要。传统方法（包括循环神经网络或Transformer模型）通过强大的神经架构引入归纳偏置来捕获复杂模式，但由于其离散特性，在推广至连续时间数据范式时存在局限性。尽管神经常微分方程（Neural ODE）及其变体在处理不规则时间序列方面展现出潜力，但它们往往难以捕捉这些序列内部的复杂关联。如何同时建模输入数据点之间的关联关系并捕获连续时间系统的动态变化，既是挑战也是迫切需求。为解决此问题，我们提出ContiFormer——将普通Transformer的关联建模能力拓展至连续时间域，通过融合Neural ODE的连续动态建模能力与Transformer的注意力机制实现显式建模。我们从数学角度刻画了ContiFormer的表达能力，并证明：通过精心设计的函数假设，诸多专用于不规则时间序列建模的Transformer变体均可被视作ContiFormer的特例。在合成数据集与真实数据集上的广泛实验表明，ContiFormer对不规则时间序列数据具有卓越的建模能力与预测性能。项目链接：https://seqml.github.io/contiformer/。