Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.

翻译：半结构化表格无处不在。已有多种任务旨在自动解释、增强和查询表格。当前方法通常需要在表格上预训练或采用特殊模型架构设计、局限于特定表格类型、或对表格和任务作出简化假设。本文迈出了开发开源大语言模型作为多样化表格任务通用模型的第一步。为此，我们构建了TableInstruct这一包含多种真实表格和任务的新数据集，用于指令微调和评估大语言模型。我们进一步开发了首个开源表格通用模型TableLlama，通过使用LongLoRA微调Llama 2（7B）以应对长上下文挑战。我们在域内和域外两种设置下进行实验。在8个域内任务中，有7个任务上TableLlama实现了与各任务当前最优方法相当或更优的性能，尽管后者通常具有任务特定设计。在6个域外数据集上，与基础模型相比，它取得了5-44个绝对百分点提升，表明在TableInstruct上训练增强了模型的泛化能力。我们开源了数据集和训练模型，以推动未来面向表格的开放通用模型开发工作。