We hypothesize that the absence of a standardized benchmark has allowed several fundamental pitfalls in GNN System design and evaluation that the community has overlooked. In this work, we propose GNNBench, a plug-and-play benchmarking platform focused on system innovation. GNNBench presents a new protocol to exchange their captive tensor data, supports custom classes in System APIs, and allows automatic integration of the same system module to many deep learning frameworks, such as PyTorch and TensorFlow. To demonstrate the importance of such a benchmark framework, we integrated several GNN systems. Our results show that integration with GNNBench helped us identify several measurement issues that deserve attention from the community.

翻译：我们假设，缺乏标准化的基准测试导致GNN系统设计与评估中存在若干社区长期忽视的根本性缺陷。本文提出GNNBench——一个面向系统创新的即插即用型基准测试平台。该平台提出新型协议以交换受控张量数据，支持系统API中的自定义类，并允许同一系统模块自动集成至PyTorch、TensorFlow等深度学习框架。为证明此类基准测试框架的重要性，我们集成了多个GNN系统。结果表明，通过GNNBench的集成，我们识别出若干亟需社区关注的测量问题。