We present RelBench, a public benchmark for solving predictive tasks over relational databases with graph neural networks. RelBench provides databases and tasks spanning diverse domains and scales, and is intended to be a foundational infrastructure for future research. We use RelBench to conduct the first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024), which combines graph neural network predictive models with (deep) tabular models that extract initial entity-level representations from raw tables. End-to-end learned RDL models fully exploit the predictive signal encoded in primary-foreign key links, marking a significant shift away from the dominant paradigm of manual feature engineering combined with tabular models. To thoroughly evaluate RDL against this prior gold-standard, we conduct an in-depth user study where an experienced data scientist manually engineers features for each task. In this study, RDL learns better models whilst reducing human work needed by more than an order of magnitude. This demonstrates the power of deep learning for solving predictive tasks over relational databases, opening up many new research opportunities enabled by RelBench.

翻译：我们提出了RelBench，这是一个用于通过图神经网络解决关系型数据库预测任务的公共基准测试框架。RelBench提供了涵盖多个领域和不同规模的数据库及任务，旨在成为未来研究的基础设施。我们利用RelBench对关系深度学习（RDL）（Fey等人，2024年）进行了首次全面研究，该方法将图神经网络预测模型与从原始表格中提取初始实体级表示的（深度）表格模型相结合。端到端学习的RDL模型充分利用了主键-外键链接中编码的预测信号，标志着从手动特征工程结合表格模型的主流范式向新范式的重大转变。为了全面评估RDL相对于先前黄金标准的性能，我们开展了深入的用户研究，由经验丰富的数据科学家为每个任务手动设计特征。在此研究中，RDL在将所需人工工作量降低一个数量级以上的同时，学习到了更优的模型。这证明了深度学习在解决关系型数据库预测任务方面的强大能力，并为RelBench所启用的众多新研究机会开辟了道路。