While existing time series foundation models primarily rely on large-scale unimodal pretraining, they lack complementary modalities to enhance time series understanding. Building multimodal foundation models is a natural next step, but it faces key challenges: 1) lack of a unified multimodal pretraining paradigm and large-scale multimodal corpora for time series analysis; 2) how to effectively integrate heterogeneous modalities and enhance model generalization. To address these challenges, we take an early step toward multimodal foundation models for time series analysis. We first propose a multimodal pretraining paradigm that leverages time series with endogenous modalities (derived images and text) and exogenous knowledge (real-world news), providing a comprehensive multi-view perspective for time series analysis. To support this, we develop an automated data construction pipeline to curate MM-TS, the first large-scale multimodal time series dataset spanning six domains, with up to one billion points. Then we propose HORAI, a frequency-enhanced multimodal foundation model. It integrates two core components: the Frequency-enhanced Cross-Modality Encoder and the Time-Frequency Decoder, designed to effectively fuse multimodal features and enhance model generalization across modalities and domains. After pretraining on MM-TS, HORAI achieves state-of-the-art zero-shot performance on time series forecasting and anomaly detection tasks, demonstrating strong generalization.

翻译：尽管现有的时间序列基础模型主要依赖于大规模单模态预训练，但它们缺乏互补模态来增强对时间序列的理解。构建多模态基础模型是自然的下一步，但它面临关键挑战：1）缺乏统一的多模态预训练范式及适用于时间序列分析的大规模多模态语料库；2）如何有效整合异构模态并增强模型泛化能力。为应对这些挑战，我们向时间序列分析的多模态基础模型迈出了早期探索的一步。我们首先提出了一种多模态预训练范式，该范式利用时间序列的内生模态（衍生图像与文本）与外生知识（真实世界新闻），为时间序列分析提供全面的多视角框架。为支持此范式，我们开发了一套自动化数据构建流程，创建了MM-TS——首个横跨六个领域、包含高达十亿数据点的大规模多模态时间序列数据集。随后，我们提出了HORAI，一种频率增强的多模态基础模型。它整合了两个核心组件：频率增强的跨模态编码器与时频解码器，旨在有效融合多模态特征并增强模型在跨模态与跨领域的泛化能力。在MM-TS上进行预训练后，HORAI在时间序列预测与异常检测任务上实现了最先进的零样本性能，展现出强大的泛化能力。