Data management has traditionally relied on synthetic data generators to generate structured benchmarks, like the TPC suite, where we can control important parameters like data size and its distribution precisely. These benchmarks were central to the success and adoption of database management systems. But more and more, data management problems are of a semantic nature. An important example is finding tables that can be unioned. While any two tables with the same cardinality can be unioned, table union search is the problem of finding tables whose union is semantically coherent. Semantic problems cannot be benchmarked using synthetic data. Our current methods for creating benchmarks involve the manual curation and labeling of real data. These methods are not robust or scalable and perhaps more importantly, it is not clear how robust the created benchmarks are. We propose to use generative AI models to create structured data benchmarks for table union search. We present a novel method for using generative models to create tables with specified properties. Using this method, we create a new benchmark containing pairs of tables that are both unionable and non-unionable but related. We thoroughly evaluate recent existing table union search methods over existing benchmarks and our new benchmark. We also present and evaluate a new table search methods based on recent large language models over all benchmarks. We show that the new benchmark is more challenging for all methods than hand-curated benchmarks, specifically, the top-performing method achieves a Mean Average Precision of around 60%, over 30% less than its performance on existing manually created benchmarks. We examine why this is the case and show that the new benchmark permits more detailed analysis of methods, including a study of both false positives and false negatives that were not possible with existing benchmarks.

翻译：数据管理传统上依赖合成数据生成器来创建结构化基准，例如TPC套件，其中我们可以精确控制数据规模及其分布等重要参数。这些基准对于数据库管理系统的成功和采用至关重要。但越来越多的数据管理问题具有语义性质。一个重要示例是寻找可以联合的表格。虽然任何具有相同基数的两个表格都可以联合，但表格联合搜索旨在寻找联合后语义上连贯的表格。语义问题无法使用合成数据进行基准测试。我们目前创建基准的方法涉及对真实数据进行人工整理和标注。这些方法既不稳健也不可扩展，而且更重要的是，尚不清楚所创建的基准的稳健性如何。我们提议使用生成式AI模型为表格联合搜索创建结构化数据基准。我们提出了一种使用生成模型创建具有指定属性表格的新方法。利用该方法，我们创建了一个新基准，其中包含可联合和不可联合但相关的表格对。我们全面评估了现有的表格联合搜索方法在现有基准及新基准上的表现。同时，我们还提出并评估了一种基于近期大型语言模型的新型表格搜索方法，覆盖所有基准。结果表明，对于所有方法而言，新基准比人工整理的基准更具挑战性：具体而言，表现最佳的方法的平均精度均值约为60%，相比其在现有手工创建基准上的表现降低了30%以上。我们探讨了这一现象的原因，并证明新基准允许对方法进行更详细的分析，包括对现有基准中无法实现的假阳性和假阴性案例的研究。