Benefiting from high-quality datasets and standardized evaluation metrics, machine learning (ML) has achieved sustained progress and widespread applications. However, while applying machine learning to relational databases (RDBs), the absence of a well-established benchmark remains a significant obstacle to the development of ML. To address this issue, we introduce ML Benchmark For Relational Databases (RDBench), a standardized benchmark that aims to promote reproducible ML research on RDBs that include multiple tables. RDBench offers diverse RDB datasets of varying scales, domains, and relational structures, organized into 4 levels. Notably, to simplify the adoption of RDBench for diverse ML domains, for any given database, RDBench exposes three types of interfaces including tabular data, homogeneous graphs, and heterogeneous graphs, sharing the same underlying task definition. For the first time, RDBench enables meaningful comparisons between ML methods from diverse domains, ranging from XGBoost to Graph Neural Networks, under RDB prediction tasks. We design multiple classification and regression tasks for each RDB dataset and report averaged results over the same dataset, further enhancing the robustness of the experimental findings. RDBench is implemented with DBGym, a user-friendly platform for ML research and application on databases, enabling benchmarking new ML methods with RDBench at ease.

翻译：受益于高质量数据集和标准化评估指标，机器学习在持续进步的同时也得到了广泛应用。然而，将机器学习应用于关系数据库时，缺乏完善的基准仍是阻碍其发展的重大障碍。为解决这一问题，我们提出了关系数据库机器学习基准（RDBench）——一个旨在促进多表关系数据库可重复机器学习研究的标准化基准。RDBench提供涵盖不同规模、领域和关系结构的多样化关系数据库数据集，并按4个层级组织。值得注意的是，为简化RDBench在不同机器学习领域的应用，对于任意给定数据库，RDBench提供三种接口类型：表格数据、同构图和异构图，它们共享相同的底层任务定义。首次地，RDBench能够实现从XGBoost到图神经网络等不同领域机器学习方法在关系数据库预测任务中的有意义比较。我们为每个关系数据库数据集设计多项分类与回归任务，并在同一数据集上报告平均结果，进一步增强了实验结果的鲁棒性。RDBench基于DBGym实现——一个面向数据库机器学习研究与应用的友好型平台，可轻松使用RDBench对新型机器学习方法进行基准测试。