Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical scales to high-dimensional vectors, we propose a numerically multi-scaled embedding module enumerating all possible scales for the scalar values. The model undergoes pretraining using the proposed numerically multi-scaled embedding with a simple contrastive objective on a large-scale dataset containing over a million sequences. We study its transfer performance on a number of univariate and multivariate classification benchmarks. Our method exhibits remarkable improvement against previous representation learning approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods.

翻译：近期关于时间序列自监督模型的研究在学习语义表征方面展现了巨大潜力，但其应用仍局限于数千条时序序列的小规模数据集。本研究针对时间序列数据的数值特性提出关键技术改进，使模型能够扩展至百万级时序序列的大规模数据集。我们采用Transformer架构，首先将输入数据划分为非重叠窗口，每个窗口通过归一化形状及其内部均值和标准差两个标量值进行表征。为将可能具有任意数值尺度的标量值嵌入高维向量，我们提出数值化多尺度嵌入模块，枚举标量值的全部潜在尺度。该模型在包含超百万序列的大规模数据集上，采用所提出的数值化多尺度嵌入方法结合简单对比目标进行预训练。我们在多个单变量与多变量分类基准上评估其迁移性能，结果表明本方法相较于现有表征学习方法取得显著提升，甚至超越了特定领域的非学习方法，创下新的最优性能记录。