The recent breakthrough of Transformers in deep learning has drawn significant attention of the time series community due to their ability to capture long-range dependencies. However, like other deep learning models, Transformers face limitations in time series prediction, including insufficient temporal understanding, generalization challenges, and data shift issues for the domains with limited data. Additionally, addressing the issue of catastrophic forgetting, where models forget previously learned information when exposed to new data, is another critical aspect that requires attention in enhancing the robustness of Transformers for time series tasks. To address these limitations, in this paper, we pre-train the time series Transformer model on a source domain with sufficient data and fine-tune it on the target domain with limited data. We introduce the \emph{One-step fine-tuning} approach, adding some percentage of source domain data to the target domains, providing the model with diverse time series instances. We then fine-tune the pre-trained model using a gradual unfreezing technique. This helps enhance the model's performance in time series prediction for domains with limited data. Extensive experimental results on two real-world datasets show that our approach improves over the state-of-the-art baselines by 4.35% and 11.54% for indoor temperature and wind power prediction, respectively.

翻译：Transformer在深度学习中的近期突破因其捕捉长程依赖关系的能力而引起了时间序列领域的广泛关注。然而，与其他深度学习模型类似，Transformer在时间序列预测中也面临局限性，包括对数据量有限领域的时间理解不足、泛化困难以及数据偏移问题。此外，解决灾难性遗忘问题（即模型在接触新数据时遗忘先前学习的信息）是增强Transformer在时间序列任务中鲁棒性的另一个关键方面。针对这些局限，本文在数据充足的源域上预训练时间序列Transformer模型，并在数据有限的目标域上进行微调。我们提出单步微调方法，将一定比例的源域数据加入目标域，为模型提供多样化的时间序列实例。随后采用逐步解冻技术对预训练模型进行微调，这有助于提升模型在数据有限领域的时间序列预测性能。在两个真实数据集上的大量实验结果表明，我们的方法在室内温度预测和风电功率预测任务上分别较现有最优基线提升了4.35%和11.54%。