We demonstrate that, for a range of state-of-the-art machine learning algorithms, the differences in generalisation performance obtained using default parameter settings and using parameters tuned via cross-validation can be similar in magnitude to the differences in performance observed between state-of-the-art and uncompetitive learning systems. This means that fair and rigorous evaluation of new learning algorithms requires performance comparison against benchmark methods with best-practice model selection procedures, rather than using default parameter settings. We investigate the sensitivity of three key machine learning algorithms (support vector machine, random forest and rotation forest) to their default parameter settings, and provide guidance on determining sensible default parameter values for implementations of these algorithms. We also conduct an experimental comparison of these three algorithms on 121 classification problems and find that, perhaps surprisingly, rotation forest is significantly more accurate on average than both random forest and a support vector machine.

翻译：我们证明,对于一系列最先进的机器学习算法,使用默认参数设置和通过交叉校验调整的参数获得的通用性能差异在规模上可以与在最先进和不具有竞争力的学习系统之间观察到的性能差异相类似。这意味着,对新的学习算法进行公平和严格的评估,要求与采用最佳做法模式选择程序的基准方法相比进行业绩比较,而不是使用默认参数设置。我们调查了三种关键机器学习算法(支持矢量机、随机森林和旋转森林)对其默认参数设置的敏感性,并指导如何确定实施这些算法的合理默认参数值。我们还就121个分类问题对这三种算法进行了实验性比较,并发现,也许令人惊讶的是,旋转森林平均比随机森林和辅助矢量机要准确得多。