Text-to-SQL benchmarks have traditionally only tested simple data access as a translation task of natural language to SQL queries. But in reality, users tend to ask diverse questions that require more complex responses including data-driven predictions or recommendations. Using the business domain as a motivating example, we introduce CORGI, a new benchmark that expands text-to-SQL to reflect practical database queries encountered by end users. CORGI is composed of synthetic databases inspired by enterprises such as DoorDash, Airbnb, and Lululemon. It provides questions across four increasingly complicated categories of business queries: descriptive, explanatory, predictive, and recommendational. This challenge calls for causal reasoning, temporal forecasting, and strategic recommendation, reflecting multi-level and multi-step agentic intelligence. We find that LLM performance degrades on higher-level questions as question complexity increases. CORGI also introduces and encourages the text-to-SQL community to consider new automatic methods for evaluating open-ended, qualitative responses in data access tasks. Our experiments show that LLMs exhibit an average 33.12% lower success execution rate (SER) on CORGI compared to existing benchmarks such as BIRD, highlighting the substantially higher complexity of real-world business needs. We release the CORGI dataset, an evaluation framework, and a submission website to support future research.

翻译：传统的文本到 SQL 基准测试仅将自然语言到 SQL 查询的转换视为简单的数据访问任务进行测试。然而在现实中，用户倾向于提出多样化的问题，这些问题需要更复杂的响应，包括数据驱动的预测或推荐。以商业领域为激励案例，我们引入了 CORGI，这是一个新的基准测试，它扩展了文本到 SQL 的范围，以反映最终用户遇到的实际数据库查询。CORGI 由受 DoorDash、Airbnb 和 Lululemon 等企业启发的合成数据库构成。它提供了跨越四个复杂度递增的商业查询类别的问题：描述性、解释性、预测性和推荐性。这一挑战需要因果推理、时间序列预测和战略性推荐，反映了多层次、多步骤的智能体智能。我们发现，随着问题复杂度的增加，大型语言模型在更高层次问题上的性能会下降。CORGI 还引入并鼓励文本到 SQL 社区考虑新的自动评估方法，以用于评估数据访问任务中开放式、定性的响应。我们的实验表明，与 BIRD 等现有基准相比，大型语言模型在 CORGI 上的平均成功执行率降低了 33.12%，突显了现实世界商业需求的复杂性显著更高。我们发布了 CORGI 数据集、评估框架和提交网站，以支持未来的研究。