Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.

翻译：大语言模型（LLMs）已成为文本到SQL任务的新范式。然而，缺乏系统性基准测试阻碍了设计高效、经济且有效的基于大语言模型的文本到SQL解决方案。为解决这一挑战，本文首先对现有提示工程方法（包括问题表征、示例选择和示例组织）进行系统全面的比较，并通过实验结果阐述其优缺点。基于这些发现，我们提出新型集成方案DAIL-SQL，该方案以86.6%的执行准确率刷新Spider排行榜并树立新标杆。为探索开源大语言模型的潜力，我们考察其在多种场景下的表现，并通过监督微调进一步提升性能。本研究既揭示了开源大语言模型在文本到SQL领域的潜力，也阐明了监督微调的优劣之处。此外，针对高效经济的基于大语言模型的文本到SQL解决方案，我们强调提示工程中的令牌效率，并在此指标下比较现有研究。希望本研究能深化对基于大语言模型的文本到SQL任务的理解，并推动后续研究及广泛应用的开展。