Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.

翻译：大型语言模型（LLMs）已成为文本到SQL任务的新范式。然而，缺乏系统化基准评估阻碍了设计高效、有效且经济的大模型文本到SQL解决方案。为应对这一挑战，本文首先对现有提示工程方法（包括问题表示、示例选择与示例组织）进行了系统化且全面的比较，并根据实验结果分析了其优缺点。基于这些发现，我们提出了一种名为DAIL-SQL的新型集成方案，该方案以86.6%的执行准确率刷新了Spider排行榜，树立了新的标杆。为探索开源大模型的潜力，我们在多种场景下对其进行了研究，并通过有监督微调进一步提升了其性能。我们的探索揭示了开源大模型在文本到SQL任务中的潜力，以及有监督微调的优缺点。此外，针对高效且经济的基于大模型的文本到SQL解决方案，我们强调了提示工程中的令牌效率，并在此指标下对先前研究成果进行了比较。我们期望本工作能增进对基于大模型的文本到SQL任务的理解，并推动其进一步研究及广泛应用。