Natural language to SQL (NL2SQL) translation enables non-expert users to query relational databases through natural language. Recently, NL2SQL agents, powered by the reasoning capabilities of Large Language Models (LLMs), have significantly advanced NL2SQL translation. Nonetheless, NL2SQL agents still make mistakes when faced with large-scale real-world databases because they lack knowledge of how to correctly leverage the underlying data (e.g., knowledge about the intent of each column) and form misconceptions about the data when querying it, leading to errors. Prior work has studied generating facts about the database to provide more context to NL2SQL agents, but such approaches simply restate database contents without addressing the agent's misconceptions. In this paper, we propose Tk-Boost, a bolt-on framework for augmenting any NL2SQL agent with tribal knowledge: knowledge that corrects the agent's misconceptions in querying the database accumulated through experience using the database. To accumulate experience, Tk-Boost first asks the NL2SQL agent to answer a few queries on the database, identifies the agent's misconceptions by analyzing its mistakes on the database, and generates tribal knowledge to address them. To enable accurate retrieval, Tk-Boost indexes this knowledge with applicability conditions that specify the query features for which the knowledge is useful. When answering new queries, Tk-Boost uses this knowledge to provide feedback to the NL2SQL agent, resolving the agent's misconceptions during SQL generation, and thus improving the agent's accuracy. Extensive experiments across the BIRD and Spider 2.0 benchmarks with various NL2SQL agents shows Tk-Boost improves NL2SQL agents accuracy by up to 16.9% on Spider 2.0 and 13.7% on BIRD
翻译:自然语言到SQL(NL2SQL)翻译使非专业用户能够通过自然语言查询关系型数据库。近年来,依托大语言模型(LLMs)推理能力的NL2SQL智能体显著推动了NL2SQL翻译的发展。然而,面对大规模现实世界数据库时,NL2SQL智能体仍会因缺乏正确利用底层数据的知识(例如对字段设计意图的理解)而在查询过程中形成错误认知,导致生成错误SQL。现有研究尝试通过生成数据库事实描述为NL2SQL智能体提供上下文,但这类方法仅复述数据库内容而未修正智能体的认知偏差。本文提出Tk-Boost——一个可无缝集成任何NL2SQL智能体的增强框架,通过注入领域经验知识来修正智能体在使用数据库过程中积累的查询认知偏差。为积累经验,Tk-Boost首先要求NL2SQL智能体对数据库执行若干查询,通过分析其在数据库上的错误识别认知偏差,并生成针对性领域经验知识进行修正。为实现精准检索,Tk-Boost为知识建立包含适用性条件的索引机制,明确界定知识适用的查询特征。在处理新查询时,Tk-Boost运用该知识体系为NL2SQL智能体提供实时反馈,在SQL生成过程中纠正其认知偏差,从而提升查询准确率。在BIRD和Spider 2.0基准测试中,针对多种NL2SQL智能体的大规模实验表明:Tk-Boost将NL2SQL智能体准确率最高提升16.9%(Spider 2.0)和13.7%(BIRD)。