Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Code-LLaMA architecture, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 14 out of 18 evaluated datasets and establishes new SoTA achievements on 7 SKG tasks. Furthermore, StructLM demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35\% and Flan-UL2 20B by an average of 10\%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.

翻译：摘要：结构化数据源（如表格、图表和数据库）是普遍存在的知识来源。尽管大语言模型（LLMs）在纯文本处理上展现出卓越能力，但其在解释和利用结构化数据方面的熟练程度仍显不足。本研究表明，LLMs在处理结构化数据时存在显著缺陷——例如，ChatGPT相比当前最优（SoTA）模型平均落后35%。为增强LLMs的结构化知识接地（SKG）能力，我们开发了包含110万示例的综合指令调优数据集。基于此数据集，我们以Code-LLaMA架构为基础，训练了参数量从7B到34B的StructLM系列模型。在18个评估数据集中，StructLM系列在14个上超越专用任务模型，并在7项SKG任务中创下新的SoTA记录。此外，StructLM在6项全新留出SKG任务上展现出强泛化能力，平均性能超过TableLlama 35%、超过Flan-UL2 20B 10%。与预期相反，我们发现扩展模型规模带来的边际收益有限——StructLM-34B相比StructLM-7B的提升微乎其微。这表明结构化知识接地仍是具有挑战性的任务，需要更具创新性的设计才能迈上新台阶。