Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.

翻译：执行准确率（EX）作为评估自然语言转SQL（NL2SQL）方案效能的广泛使用指标，正日益变得不可靠。该指标对语法变化敏感，忽略了问题可能存在多种解释，且容易被错误的标准SQL误导。为解决这一问题，我们提出ROSE——一种以意图为中心的评估指标，其核心在于判定预测SQL是否真正回答了问题，而非在依赖参考基准范式下与标准SQL的一致性。ROSE采用对抗性的"验证-反驳"级联机制：SQL验证器独立评估预测SQL针对用户意图的语义正确性，而对抗性反驳器则以标准SQL为证据，对该判断提出挑战并加以修正。在我们经专家标定的验证集ROSE-VEC上，ROSE与人类专家的一致性最高，其Cohen's Kappa系数较次优指标提升近24%。我们还对19种NL2SQL方法进行了大规模重评估，揭示了四项重要发现。为促进更可靠的NL2SQL研究，我们已公开ROSE及ROSE-VEC。