Benefiting from high-quality datasets and standardized evaluation metrics, machine learning (ML) has achieved sustained progress and widespread applications. However, while applying machine learning to relational databases (RDBs), the absence of a well-established benchmark remains a significant obstacle to the development of ML. To address this issue, we introduce ML Benchmark For Relational Databases (RDBench), a standardized benchmark that aims to promote reproducible ML research on RDBs that include multiple tables. RDBench offers diverse RDB datasets of varying scales, domains, and relational structures, organized into 4 levels. Notably, to simplify the adoption of RDBench for diverse ML domains, for any given database, RDBench exposes three types of interfaces including tabular data, homogeneous graphs, and heterogeneous graphs, sharing the same underlying task definition. For the first time, RDBench enables meaningful comparisons between ML methods from diverse domains, ranging from XGBoost to Graph Neural Networks, under RDB prediction tasks. We design multiple classification and regression tasks for each RDB dataset and report averaged results over the same dataset, further enhancing the robustness of the experimental findings. RDBench is implemented with DBGym, a user-friendly platform for ML research and application on databases, enabling benchmarking new ML methods with RDBench at ease.

翻译：依托于高质量数据集与标准化评估指标，机器学习已在诸多领域取得持续进展与广泛应用。然而，当将机器学习应用于关系数据库时，缺乏完善的基准测试仍是制约其发展的重大障碍。为解决该问题，我们提出面向关系数据库的机器学习基准测试（RDBench），旨在促进包含多表结构的关系数据库的可复现机器学习研究。RDBench 提供涵盖不同规模、领域及关系结构的多样化关系数据库数据集，并划分为四个层级。值得注意的是，为便于在不同机器学习领域采用 RDBench，对于任意给定数据库，RDBench 暴露了三种接口（表格数据、同构图与异构图），且共享同一底层任务定义。首次，RDBench 使得从 XGBoost 到图神经网络等不同领域的机器学习方法在关系数据库预测任务中得以进行有意义的比较。我们为每个关系数据库数据集设计了多个分类与回归任务，并在同一数据集上报告平均结果，从而进一步增强实验发现的稳健性。RDBench 基于 DBGym（一款用户友好的数据库机器学习研究与应用程序平台）实现，可轻松利用 RDBench 对新机器学习方法进行基准测试。