Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborates their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. Towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. Additionally, we investigate open-source LLMs in in-context learning, and further enhance their performance with task-specific supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the task-specific supervised fine-tuning. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspire further investigations and broad applications.

翻译：大型语言模型（LLMs）已成为文本到SQL任务的新范式。然而，系统性基准的缺失制约了设计高效、经济且基于LLM的文本到SQL解决方案的发展。为应对这一挑战，本文首先对现有提示工程方法（包括问题表示、示例选择和示例组织）进行了系统而广泛的比较，并根据实验结果阐明其优劣。基于这些发现，我们提出了一种名为DAIL-SQL的新型集成解决方案，以86.6%的执行准确率刷新了Spider排行榜，树立了新标杆。为构建高效经济的LLM文本到SQL方案，我们强调提示工程中的令牌效率，并在该指标下对比了先前研究。此外，我们探究了开源LLM在上下文学习中的表现，并通过任务特定监督微调进一步提升了其性能。我们的探索凸显了开源LLM在文本到SQL中的潜力，以及任务特定监督微调的利与弊。期望我们的工作能深化对基于LLM的文本到SQL的理解，并推动相关研究及广泛应用。