We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

翻译：我们提出了一种基于Transformer的高效设计，用于多元时间序列预测和自监督表示学习。该设计基于两个关键组件：（i）将时间序列分割为子序列级别的片段，这些片段作为输入令牌输入到Transformer；（ii）通道独立性，其中每个通道包含一个单变量时间序列，该序列在所有序列间共享相同的嵌入和Transformer权重。片段化设计自然具有三重优势：局部语义信息保留在嵌入中；在相同回溯窗口下，注意力图的计算和内存使用量呈二次方减少；模型可以处理更长的历史数据。我们的通道独立片段时间序列Transformer（PatchTST）与最先进的基于Transformer的模型相比，可显著提高长期预测的准确性。我们还将该模型应用于自监督预训练任务，并获得了出色的微调性能，其在大数据集上优于有监督训练。将一个数据集上的掩码预训练表示迁移到其他数据集也能产生最先进的预测准确性。代码可在以下网址获取：https://github.com/yuqinie98/PatchTST。