The task of information extraction (IE) is to extract structured knowledge from text. However, it is often not straightforward to utilize IE output due to the mismatch between the IE ontology and the downstream application needs. We propose a new formulation of IE TEXT2DB that emphasizes the integration of IE output and the target database (or knowledge base). Given a user instruction, a document set, and a database, our task requires the model to update the database with values from the document set to satisfy the user instruction. This task requires understanding user instructions for what to extract and adapting to the given DB/KB schema for how to extract on the fly. To evaluate this new task, we introduce a new benchmark featuring common demands such as data infilling, row population, and column addition. In addition, we propose an LLM agent framework OPAL (Observe-PlanAnalyze LLM) which includes an Observer component that interacts with the database, the Planner component that generates a code-based plan with calls to IE models, and the Analyzer component that provides feedback regarding code quality before execution. Experiments show that OPAL can successfully adapt to diverse database schemas by generating different code plans and calling the required IE models. We also highlight difficult cases such as dealing with large databases with complex dependencies and extraction hallucination, which we believe deserve further investigation. Source code: https://github.com/yzjiao/Text2DB

翻译：信息抽取（IE）任务旨在从文本中提取结构化知识。然而，由于IE本体与下游应用需求之间的不匹配，IE输出往往难以直接利用。我们提出了一种新的IE任务形式化方法TEXT2DB，强调IE输出与目标数据库（或知识库）的集成。给定用户指令、文档集和数据库，该任务要求模型根据文档集更新数据库中的值以满足用户指令。此任务需要理解用户指令以确定提取内容，并动态适应给定的数据库/知识库模式以确定提取方式。为评估这一新任务，我们引入了一个包含数据填充、行数据补全和列添加等常见需求的新基准。此外，我们提出了一个基于大语言模型（LLM）的代理框架OPAL（观察-规划-分析LLM），该框架包含一个与数据库交互的观察器组件、一个生成基于代码的计划并调用IE模型的规划器组件，以及一个在执行前提供代码质量反馈的分析器组件。实验表明，OPAL能够通过生成不同的代码计划并调用所需的IE模型，成功适应多样化的数据库模式。我们还指出了处理具有复杂依赖关系的大型数据库和提取幻觉等困难案例，这些我们认为值得进一步研究。源代码：https://github.com/yzjiao/Text2DB