Large language models are being rapidly deployed across many fields such as healthcare, finance, transportation, and energy, where time-series data are fundamental components. The current works are still limited in their ability to perform reasoning that involves both time-series and the corresponding textual content. We address this gap by introducing Chat-TS, a large language model (LLM) based framework designed to support reasoning over time series and textual data. Unlike traditional models, Chat-TS integrates time-series tokens into LLMs' vocabulary, enhancing its reasoning ability over both modalities without compromising core natural language capabilities. To support learning and evaluation, we contribute new datasets: the TS Instruct Training Dataset (pairing diverse time-series data with relevant text instructions and responses for instruction tuning), the TS Instruct Question and Answer (QA) Gold Dataset (multiple-choice questions to evaluate multimodal reasoning), and a TS Instruct Quantitative Probing Set (a small subset of TS Instruct QA reasoning tasks alongside math and decision-making questions for LLM evaluation). We design a training strategy to preserve the inherent reasoning capabilities of LLMs while augmenting them for time-series reasoning. Experiments show that Chat-TS achieves state-of-the-art performance in multimodal reasoning tasks by maintaining strong natural language proficiency while improving time-series reasoning.

翻译：大型语言模型正迅速部署于医疗、金融、交通和能源等多个领域，这些领域以时序数据为基础要素。现有研究在处理同时涉及时序数据与对应文本内容的推理任务时仍存在局限。为填补这一空白，本文提出Chat-TS——一个基于大型语言模型的框架，旨在支持对时序数据与文本数据的联合推理。与传统模型不同，Chat-TS将时序数据标记集成至大型语言模型的词汇表中，在保持核心自然语言处理能力的同时，增强其对双模态数据的推理能力。为支持训练与评估，我们构建了以下新数据集：TS Instruct训练数据集（将多样化时序数据与相关文本指令及响应配对，用于指令微调）、TS Instruct问答黄金数据集（通过多选题评估多模态推理能力）以及TS Instruct定量探测集（包含少量TS Instruct问答推理任务及数学与决策问题，用于大型语言模型评估）。我们设计了专门的训练策略，在保持大型语言模型固有推理能力的同时增强其时序推理能力。实验表明，Chat-TS在保持强大自然语言处理能力的基础上显著提升了时序推理性能，在多模态推理任务中达到了最先进的水平。