Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 6-48 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We will open-source our dataset and trained model to boost future work on developing open generalist models for tables.

翻译：半结构化表格无处不在。已有多种任务旨在自动解读、扩充和查询表格。当前方法通常需要在表格数据上预训练或设计特殊模型架构，且局限于特定表格类型，或对表格和任务做出简化假设。本文首次探索将开源大型语言模型（LLM）发展为可处理多种表格任务的通用模型。为此，我们构建了TableInstruct数据集，包含多样化的真实表格与任务，用于指令微调和评估LLM。我们进一步开发了首个开源表格通用模型TableLlama，通过使用LongLoRA微调Llama 2（7B）以应对长上下文挑战。我们在领域内和领域外两种设置下进行实验。在8个领域内任务中，TableLlama在7个任务上达到或超越各任务当前最优方法（SOTA）的性能表现，尽管后者常具有任务特定设计。在6个领域外数据集上，相较于基线模型，它实现了6-48个绝对百分点提升，表明在TableInstruct上训练增强了模型的泛化能力。我们将开源数据集和训练模型，以推动未来面向表格的开源通用模型研究。