Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge. However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors. This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models. While BIRD-Bench was created to model dirty and noisy database values, it was not created to contain noise and errors in the questions and gold queries. We found that noise in questions and gold queries are prevalent in the dataset, with varying amounts across domains, and with an uneven distribution between noise types. The presence of incorrect gold SQL queries, which then generate incorrect gold answers, has a significant impact on the benchmark's reliability. Surprisingly, when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods. We conclude that informative noise labels and reliable benchmarks are crucial to developing new Text-to-SQL methods that can handle varying types of noise.

翻译：文本到SQL（Text-to-SQL）涉及将自然语言转换为结构化查询语言（SQL），对于使非专业人员无需专业知识即可广泛访问结构化数据库至关重要。然而，由于包含歧义问题和语法错误等“噪声”的多种因素，为此类任务设计模型颇具挑战性。本研究深入分析了广泛使用的BIRD-Bench基准测试中噪声的分布与类型及其对模型的影响。尽管BIRD-Bench旨在模拟脏乱和含噪的数据库值，但其设计初衷并非在问题和黄金查询中引入噪声和错误。我们发现，数据集中问题和黄金查询中的噪声普遍存在，不同领域间的噪声量各异，且噪声类型的分布不均衡。错误的黄金SQL查询（进而生成错误的黄金答案）对基准测试的可靠性产生了显著影响。令人惊讶的是，当基于修正后的SQL查询评估模型时，零样本基线方法的表现超越了最先进的提示方法。我们得出结论：提供信息丰富的噪声标签和可靠的基准测试对于开发能够处理各类噪声的新型文本到SQL方法至关重要。