Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.

翻译：大型语言模型（LLMs）已成为文本到SQL任务的新范式。然而，缺乏系统性基准阻碍了设计有效、高效且经济的基于LLM的文本到SQL解决方案的发展。为解决这一挑战，本文首先对现有提示工程方法（包括问题表示、示例选择和示例组织）进行系统且广泛的比较，并结合实验结果阐述其优劣。基于这些发现，我们提出一种名为DAIL-SQL的新型集成解决方案，以86.6%的执行准确率刷新Spider排行榜并树立新标杆。为探索开源LLM的潜力，我们在多种场景下对其进行研究，并通过有监督微调进一步提升其性能。我们的探索揭示了开源LLM在文本到SQL任务中的潜力，以及有监督微调的优缺点。此外，为构建高效且经济的基于LLM的文本到SQL解决方案，我们强调提示工程中的令牌效率，并在此指标下比较了先前研究。我们希望本研究能加深对基于LLM的文本到SQL任务的理解，并推动进一步的研究与广泛应用。