Large pre-trained models for zero/few-shot learning excel in language and vision domains but encounter challenges in multivariate time series (TS) due to the diverse nature and scarcity of publicly available pre-training data. Consequently, there has been a recent surge in utilizing pre-trained large language models (LLMs) with token adaptations for TS forecasting. These approaches employ cross-domain transfer learning and surprisingly yield impressive results. However, these models are typically very slow and large (~billion parameters) and do not consider cross-channel correlations. To address this, we present Tiny Time Mixers (TTM), a significantly small model based on the lightweight TSMixer architecture. TTM marks the first success in developing fast and tiny general pre-trained models (<1M parameters), exclusively trained on public TS datasets, with effective transfer learning capabilities for forecasting. To tackle the complexity of pre-training on multiple datasets with varied temporal resolutions, we introduce several novel enhancements such as adaptive patching, dataset augmentation via downsampling, and resolution prefix tuning. Moreover, we employ a multi-level modeling strategy to effectively model channel correlations and infuse exogenous signals during fine-tuning, a crucial capability lacking in existing benchmarks. TTM shows significant accuracy gains (12-38\%) over popular benchmarks in few/zero-shot forecasting. It also drastically reduces the compute needs as compared to LLM-TS methods, with a 14X cut in learnable parameters, 106X less total parameters, and substantial reductions in fine-tuning (65X) and inference time (54X). In fact, TTM's zero-shot often surpasses the few-shot results in many popular benchmarks, highlighting the efficacy of our approach. Models and source code are available at https://huggingface.co/ibm/TTM

翻译：大型预训练模型在零样本/少样本学习中在语言和视觉领域表现出色，但由于多变量时间序列（TS）的多样性以及公开预训练数据的稀缺性，其在时间序列领域面临挑战。因此，近期兴起利用预训练大语言模型（LLMs）通过令牌适配进行时间序列预测的方法。这些方法采用跨域迁移学习，并意外地取得了令人印象深刻的结果。然而，这些模型通常非常缓慢且庞大（约十亿参数），且未考虑跨通道相关性。为解决此问题，我们提出小时间混合器（TTM），一种基于轻量级TSMixer架构的极小模型。TTM首次成功开发出快速且小巧的通用预训练模型（参数少于100万），完全基于公开时间序列数据集训练，并具备有效的迁移学习预测能力。为应对跨多种时间分辨率数据集预训练的复杂性，我们引入了多项新增强技术，包括自适应分块、通过降采样进行数据集增强以及分辨率前缀调优。此外，我们采用多层次建模策略，有效建模通道相关性并在微调过程中注入外源信号，这一关键能力在现有基准中缺失。TTM在少样本/零样本预测中相比流行基准实现了显著的精度提升（12-38%）。同时，与LLM-TS方法相比，它大幅降低了计算需求，可学习参数减少14倍，总参数减少106倍，微调时间（65倍）和推理时间（54倍）也显著降低。事实上，TTM的零样本预测在许多流行基准中常超越少样本结果，凸显了本方法的有效性。模型和源代码可在 https://huggingface.co/ibm/TTM 获取。