With the booming demand for machine learning applications, it has been recognized that the number of knowledgeable data scientists can not scale with the growing data volumes and application needs in our digital world. In response to this demand, several automated machine learning (AutoML) frameworks have been developed to fill the gap of human expertise by automating the process of building machine learning pipelines. Each framework comes with different heuristics-based design decisions. In this study, we present a comprehensive evaluation and comparison of the performance characteristics of six popular AutoML frameworks, namely, AutoWeka, AutoSKlearn, TPOT, Recipe, ATM, and SmartML, across 100 data sets from established AutoML benchmark suites. Our experimental evaluation considers different aspects for its comparison, including the performance impact of several design decisions, including time budget, size of search space, meta-learning, and ensemble construction. The results of our study reveal various interesting insights that can significantly guide and impact the design of AutoML frameworks.

翻译：随着机器学习应用需求的激增，人们认识到具备专业知识的数据科学家数量已无法跟上数字世界中数据量和应用需求的增长。为应对这一需求，多个自动化机器学习（AutoML）框架被开发出来，通过自动化构建机器学习流程来弥补人类专业知识的不足。每个框架都基于不同的启发式设计决策。本研究对六种主流AutoML框架（即AutoWeka、AutoSKlearn、TPOT、Recipe、ATM和SmartML）在来自已建立的AutoML基准套件的100个数据集上的性能特征进行了全面评估与比较。我们的实验评估从多个角度展开对比，包括时间预算、搜索空间规模、元学习以及集成构建等设计决策对性能的影响。研究结果揭示了多项有趣见解，这些发现将对AutoML框架的设计产生重要指导作用。