Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.

翻译：半结构化表格无处不在。已有多种任务旨在自动解释、扩充和查询表格。当前方法通常需要对表格进行预训练或采用特殊模型架构设计，局限于特定表格类型，或对表格和任务做出简化假设。本文首次探索开发开源大型语言模型（LLM）作为多样化表格任务的通用模型。为此，我们构建了TableInstruct数据集，包含多种真实表格与任务，用于指令微调与评估LLM。进一步，我们通过使用LongLoRA微调Llama 2（7B）以应对长上下文挑战，开发出首个开源表格通用模型TableLlama。我们在域内和域外两种设置下进行实验。在8个域内任务中，TableLlama在7个任务上达到或优于当前最优方法（SOTA）的同等性能，尽管后者通常具有任务特定设计。在6个域外数据集上，相比基座模型，它获得了5-44个绝对百分点的提升，表明在TableInstruct上的训练增强了模型的泛化能力。我们开源数据集及训练模型，以推动未来开放表格通用模型的开发。