Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Code-LLaMA architecture, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 14 out of 18 evaluated datasets and establishes new SoTA achievements on 7 SKG tasks. Furthermore, StructLM demonstrates exceptional generalization across 6 novel SKG tasks. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.

翻译：结构化数据源（如表格、图和数据库）是普遍存在的知识来源。尽管大语言模型在纯文本处理上展现了显著能力，但其在解释和利用结构化数据方面的熟练度仍然有限。我们的研究揭示了大语言模型在处理结构化数据方面存在显著缺陷——例如，ChatGPT相较于当前最优模型的性能平均落后35%。为增强大语言模型的结构化知识基础能力，我们开发了一个包含110万样本的综合指令调优数据集。基于该数据集，我们以Code-LLaMA架构为基础，训练了参数量从7B到34B的StructLM系列模型。在18个评估数据集的14个任务中，我们的StructLM系列超越了专有任务模型，并在7个结构化知识基础任务上创下新的最优性能记录。此外，StructLM在6个未见过的结构化知识基础任务中展现出卓越的泛化能力。与预期相反，我们发现模型规模扩展带来的收益微乎其微——StructLM-34B相较StructLM-7B仅有轻微提升。这表明结构化知识基础仍具挑战性，需要更具创新性的设计才能实现质的突破。