A long standing goal of the data management community is to develop general, automated systems that ingest semi-structured documents and output queryable tables without human effort or domain specific customization. Given the sheer variety of potential documents, state-of-the art systems make simplifying assumptions and use domain specific training. In this work, we ask whether we can maintain generality by using large language models (LLMs). LLMs, which are pretrained on broad data, can perform diverse downstream tasks simply conditioned on natural language task descriptions. We propose and evaluate EVAPORATE, a simple, prototype system powered by LLMs. We identify two fundamentally different strategies for implementing this system: prompt the LLM to directly extract values from documents or prompt the LLM to synthesize code that performs the extraction. Our evaluations show a cost-quality tradeoff between these two approaches. Code synthesis is cheap, but far less accurate than directly processing each document with the LLM. To improve quality while maintaining low cost, we propose an extended code synthesis implementation, EVAPORATE-CODE+, which achieves better quality than direct extraction. Our key insight is to generate many candidate functions and ensemble their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms the state-of-the art systems, but does so using a sublinear pass over the documents with the LLM. This equates to a 110x reduction in the number of tokens the LLM needs to process, averaged across 16 real-world evaluation settings of 10k documents each.

翻译：数据管理领域的一个长期目标是开发通用自动化系统，能够处理半结构化文档并直接生成可查询表格，无需人工干预或领域特定定制。鉴于潜在文档的多样性，现有系统通常采用简化假设并依赖领域专用训练。本文探讨利用大规模语言模型（LLMs）保持通用性的可行性——这类在广泛数据上预训练的模型仅需配合自然语言任务描述即可执行多样化下游任务。我们提出并评估了基于LLMs的简易原型系统EVAPORATE，并识别出两种根本不同的实现策略：直接提示LLM从文档中提取数值，或提示LLM合成执行提取任务的代码。实验表明两种方法存在成本-质量权衡：代码合成成本低廉，但准确性远低于逐文档直接处理。为在保持低成本的同时提升质量，我们提出扩展型代码合成实现EVAPORATE-CODE+，其质量优于直接提取。关键创新在于生成多个候选函数，并通过弱监督集成其提取结果。EVAPORATE-CODE+不仅超越现有最优系统，且仅需对文档进行次线性扫描（使用LLM处理），在16个各含10k文档的真实评估场景中，平均将LLM需处理的token数量降低110倍。