Deep learning has contributed remarkably to the advancement of time series analysis. Still, deep models can encounter performance bottlenecks in real-world data-scarce scenarios, which can be concealed due to the performance saturation with small models on current benchmarks. Meanwhile, large models have demonstrated great powers in these scenarios through large-scale pre-training. Continuous progress has been achieved with the emergence of large language models, exhibiting unprecedented abilities such as few-shot generalization, scalability, and task generality, which are however absent in small deep models. To change the status quo of training scenario-specific small models from scratch, this paper aims at the early development of large time series models (LTSM). During pre-training, we curate large-scale datasets with up to 1 billion time points, unify heterogeneous time series into single-series sequence (S3) format, and develop the GPT-style architecture toward LTSMs. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task. The outcome of this study is a Time Series Transformer (Timer), which is generative pre-trained by next token prediction and adapted to various downstream tasks with promising capabilities as an LTSM. Code and datasets are available at: https://github.com/thuml/Large-Time-Series-Model.

翻译：深度学习显著推动了时间序列分析的发展。然而，在现实世界数据稀缺的场景中，深度模型可能遭遇性能瓶颈；由于当前基准测试中小型模型已出现性能饱和，这一问题常被掩盖。与此同时，大型模型通过大规模预训练在这些场景中展现出强大能力。随着大语言模型的兴起，相关研究持续取得进展，其表现出小规模深度模型所不具备的少样本泛化、可扩展性和任务通用性等前所未有的能力。为改变当前针对特定场景从头训练小型模型的现状，本文致力于大型时间序列模型的早期开发。在预训练阶段，我们构建了包含高达10亿时间点的大规模数据集，将异构时间序列统一为单序列格式，并针对大型时间序列模型开发了GPT风格的架构。为满足多样化应用需求，我们将时间序列的预测、插补和异常检测任务统一转化为生成式任务。本研究的核心成果是时间序列Transformer模型，该模型通过下一标记预测进行生成式预训练，并能适应多种下游任务，展现出作为大型时间序列模型的潜力。代码与数据集已开源：https://github.com/thuml/Large-Time-Series-Model。