Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries. However, recent studies reveal that text-to-SQL models are vulnerable to task-specific perturbations. Previous curated robustness test sets usually focus on individual phenomena. In this paper, we propose a comprehensive robustness benchmark based on Spider, a cross-domain text-to-SQL benchmark, to diagnose the model robustness. We design 17 perturbations on databases, natural language questions, and SQL queries to measure the robustness from different angles. In order to collect more diversified natural question perturbations, we utilize large pretrained language models (PLMs) to simulate human behaviors in creating natural questions. We conduct a diagnostic study of the state-of-the-art models on the robustness set. Experimental results reveal that even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop on the most challenging perturbation. We also present a breakdown analysis regarding text-to-SQL model designs and provide insights for improving model robustness.

翻译：神经文本到SQL模型在将自然语言问题翻译为SQL查询方面取得了显著性能。然而，近期研究表明，文本到SQL模型容易受到任务特定扰动的影响。此前构建的鲁棒性测试集通常聚焦于单一现象。本文基于跨领域文本到SQL基准Spider，提出一个综合性鲁棒性基准，用于诊断模型鲁棒性。我们设计了涵盖数据库、自然语言问题和SQL查询的17种扰动，从不同角度衡量鲁棒性。为收集更多样化的自然语言问题扰动，我们利用大规模预训练语言模型模拟人类在生成自然问题时行为。我们对当前最先进模型在鲁棒性集上进行了诊断研究。实验结果表明，即使是最鲁棒的模型，在整体性能上仍下降14.0%，在最具挑战性的扰动上性能下降达50.7%。我们还针对文本到SQL模型设计进行了分解分析，为提升模型鲁棒性提供了见解。