Recent advances in large language models (LLMs) have led to significant improvements in translating natural language questions into SQL queries. While achieving high accuracy in SQL generation is crucial, little is known about the extent to which these text-to-SQL models can reliably handle diverse types of questions encountered during real-world deployment, including unanswerable ones. To explore this aspect, we introduce TrustSQL, a new benchmark designed to assess the reliability of text-to-SQL models in both single-database and cross-database settings. TrustSQL requires models to provide one of two outputs: 1) an SQL prediction or 2) abstention from making an SQL prediction, either due to potential errors in the generated SQL or when faced with unanswerable questions. For model evaluation, we explore various modeling approaches specifically designed for this task: 1) optimizing separate models for answerability detection, SQL generation, and error detection, which are then integrated into a single pipeline; and 2) developing a unified approach that uses a single model to solve this task. Experimental results using our new reliability score show that addressing this challenge involves many different areas of research and opens new avenues for model development. However, none of the methods consistently surpasses the reliability scores of a naive baseline that abstains from SQL predictions for all questions, with varying penalties.

翻译：近期大语言模型（LLMs）的进展显著提升了将自然语言问题转化为SQL查询的能力。尽管实现高精度的SQL生成至关重要，但关于这些文本到SQL模型在实际部署中（包括面对不可回答问题）能否可靠处理各类问题的能力尚不明确。为探究这一维度，我们提出TrustSQL——一个专为评估单数据库与跨数据库场景下文本到SQL模型可靠性而设计的新基准。TrustSQL要求模型提供两种输出之一：1）SQL预测结果；或2）当生成的SQL可能存在错误或遇到不可回答问题时，放弃SQL预测。在模型评估方面，我们探索了针对该任务设计的多种建模方法：1）分别优化可回答性检测、SQL生成和错误检测的独立模型，并将其整合至单一流水线；2）开发使用单一模型解决此任务的统一方法。采用我们提出的可靠性评分进行的实验表明，应对这一挑战涉及多个研究领域，并为模型开发开辟了新途径。然而，所有方法均未能持续超越对所有问题均放弃SQL预测的朴素基线的可靠性得分，且存在不同惩罚程度的差异。