Text-to-SQL benchmarks play a crucial role in evaluating the progress made in the field and the ranking of different models. However, accurately matching a model-generated SQL query to a reference SQL query in a benchmark fails for various reasons, such as underspecified natural language queries, inherent assumptions in both model-generated and reference queries, and the non-deterministic nature of SQL output under certain conditions. In this paper, we conduct an extensive study of several prominent cross-domain text-to-SQL benchmarks and re-evaluate some of the top-performing models within these benchmarks, by both manually evaluating the SQL queries and rewriting them in equivalent expressions. Our evaluation reveals that attaining a perfect performance on these benchmarks is unfeasible due to the multiple interpretations that can be derived from the provided samples. Furthermore, we find that the true performance of the models is underestimated and their relative performance changes after a re-evaluation. Most notably, our evaluation reveals a surprising discovery: a recent GPT4-based model surpasses the gold standard reference queries in the Spider benchmark in our human evaluation. This finding highlights the importance of interpreting benchmark evaluations cautiously, while also acknowledging the critical role of additional independent evaluations in driving advancements in the field.

翻译：文本到SQL基准测试在评估该领域进展和不同模型排名方面发挥着关键作用。然而，将模型生成的SQL查询与基准测试中的参考SQL查询精确匹配，可能因多种原因而失败，例如自然语言查询表述不充分、模型生成查询与参考查询中固有的假设，以及特定条件下SQL输出的非确定性特性。在本文中，我们对多个著名的跨领域文本到SQL基准测试进行了广泛研究，并通过手动评估SQL查询及其等价表达式重写，重新评估了这些基准测试中一些表现最佳的模型。我们的评估表明，由于给定样本可衍生多种解释，在这些基准测试上达到完美性能是不可行的。此外，我们发现模型的真实性能被低估，且其相对表现在重新评估后发生了变化。最值得注意的是，我们的评估揭示了一个惊人发现：在Spider基准测试的人工评估中，近期基于GPT4的模型超越了黄金标准参考查询。这一发现强调了谨慎解读基准测试评估的重要性，同时承认额外独立评估在推动领域进步中的关键作用。