Tree-boosting is a widely used machine learning technique for tabular data. However, its out-of-sample accuracy is critically dependent on multiple hyperparameters. In this article, we empirically compare several popular methods for hyperparameter optimization for tree-boosting including random grid search, the tree-structured Parzen estimator (TPE), Gaussian-process-based Bayesian optimization (GP-BO), Hyperband, the sequential model-based algorithm configuration (SMAC) method, and deterministic full grid search using $59$ regression and classification data sets. We find that the SMAC method clearly outperforms all the other considered methods. We further observe that (i) a relatively large number of trials larger than $100$ is required for accurate tuning, (ii) using default values for hyperparameters yields very inaccurate models, (iii) all considered hyperparameters can have a material effect on the accuracy of tree-boosting, i.e., there is no small set of hyperparameters that is more important than others, and (iv) choosing the number of boosting iterations using early stopping yields more accurate results compared to including it in the search space for regression tasks.

翻译：树提升是一种广泛应用于表格数据的机器学习技术。然而，其样本外精度关键依赖于多个超参数。本文通过59个回归和分类数据集，实证比较了树提升超参数优化的几种流行方法，包括随机网格搜索、树结构帕尔森估计器（TPE）、基于高斯过程的贝叶斯优化（GP-BO）、Hyperband、基于序列模型算法配置方法（SMAC）以及确定性全网格搜索。我们发现SMAC方法明显优于所有其他考虑的方法。进一步观察到：(i) 精确调优需要超过100次的相对较大试验次数；(ii) 使用默认超参数值会导致模型精度极差；(iii) 所有考虑的超参数均对树提升的精度产生实质性影响，即不存在比其他超参数更重要的小规模超参数集；(iv) 在回归任务中，采用早停法选择提升迭代次数比将其纳入搜索空间能得到更精确的结果。