A foundation model is a machine learning model trained on a large and diverse set of data, typically using self-supervised learning-based pre-training techniques, that can be adapted to various downstream tasks. However, current research on time series pre-training has mostly focused on models pre-trained solely on data from a single domain, resulting in a lack of knowledge about other types of time series. However, current research on time series pre-training has predominantly focused on models trained exclusively on data from a single domain. As a result, these models possess domain-specific knowledge that may not be easily transferable to time series from other domains. In this paper, we aim to develop an effective time series foundation model by leveraging unlabeled samples from multiple domains. To achieve this, we repurposed the publicly available UCR Archive and evaluated four existing self-supervised learning-based pre-training methods, along with a novel method, on the datasets. We tested these methods using four popular neural network architectures for time series to understand how the pre-training methods interact with different network designs. Our experimental results show that pre-training improves downstream classification tasks by enhancing the convergence of the fine-tuning process. Furthermore, we found that the proposed pre-training method, when combined with the Transformer model, outperforms the alternatives.

翻译：基础模型是一种在大规模、多样化数据集上训练的机器学习模型，通常采用基于自监督学习的预训练技术，可适应各类下游任务。然而，当前时间序列预训练研究主要聚焦于仅在单一领域数据上训练的模型，导致模型缺乏对其他类型时间序列数据的认知。实际上，现有时间序列预训练研究大多局限于单一领域数据训练的模型，使其具备领域特异性知识，难以有效迁移至其他领域的时间序列。本文旨在通过利用来自多个领域的无标签样本，构建高效的时间序列基础模型。为此，我们重新利用公开的UCR数据集档案，在数据集上评估了四种现有基于自监督学习的预训练方法及一种新方法。我们采用四种主流时间序列神经网络架构对上述方法进行测试，以探究预训练方法与不同网络设计的交互机制。实验结果表明：预训练通过加速微调过程的收敛性，显著提升了下游分类任务的性能。此外，我们发现所提出的预训练方法与Transformer模型结合时，其表现优于其他备选方案。