Benchmarking models via classical simulations is one of the main ways to judge ideas in quantum machine learning before noise-free hardware is available. However, the huge impact of the experimental design on the results, the small scales within reach today, as well as narratives influenced by the commercialisation of quantum technologies make it difficult to gain robust insights. To facilitate better decision-making we develop an open-source package based on the PennyLane software framework and use it to conduct a large-scale study that systematically tests 12 popular quantum machine learning models on 6 binary classification tasks used to create 160 individual datasets. We find that overall, out-of-the-box classical machine learning models outperform the quantum classifiers. Moreover, removing entanglement from a quantum model often results in as good or better performance, suggesting that "quantumness" may not be the crucial ingredient for the small learning tasks considered here. Our benchmarks also unlock investigations beyond simplistic leaderboard comparisons, and we identify five important questions for quantum model design that follow from our results.

翻译：通过经典模拟进行模型基准测试，是在无噪声硬件可用之前评判量子机器学习思想的主要方法之一。然而，实验设计对结果的巨大影响、当前可及的小规模研究，以及量子技术商业化带来的叙事影响，使得获取稳健见解变得困难。为促进更优决策，我们基于PennyLane软件框架开发了一个开源软件包，并利用其对6项二分类任务（用于创建160个独立数据集）中的12种流行量子机器学习模型进行系统性大规模研究。我们发现，整体而言，开箱即用的经典机器学习模型表现优于量子分类器。此外，从量子模型中移除纠缠往往能带来相同或更好的性能，这表明"量子性"可能并非此处考虑的小规模学习任务的关键要素。我们的基准测试也突破了简单的排行榜比较，根据实验结果提出了量子模型设计的五个重要问题。