The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the selection-based prompts to facilitate distribution adaptation in non-stationary time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on a number of time series benchmark datasets. This performance gain is observed not only in standard supervised learning settings but also in scenarios involving previously unseen datasets as well as in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.

翻译：过去十年，深度学习在时间序列建模领域取得了显著进展。尽管最先进的架构在不同应用和领域中表现各异，但其结果已达到顶尖水平。与此同时，在自然语言处理中，生成式预训练Transformer（GPT）通过跨多种文本数据训练通用模型，展现了卓越性能。探索GPT类架构是否能有效捕捉时间序列的内在动态属性并显著提升准确性，是一个引人关注的问题。本文提出了一种名为TEMPO的新框架，能够有效学习时间序列表示。我们专注于利用预训练模型的两个关键时间序列任务归纳偏置：（i）对趋势、季节和残差分量复杂交互作用的分解；（ii）引入基于选择的提示，以促进非平稳时间序列中的分布自适应。TEMPO扩展了对来自不同领域数据中真实世界时间现象进行动态建模的能力。实验表明，在多个时间序列基准数据集上，TEMPO的性能优于最先进方法。这一性能提升不仅体现在标准监督学习设置中，还体现在涉及未见数据集以及多模态输入的场景中。这一令人信服的发现凸显了TEMPO作为基础模型构建框架的潜力。