Machine learning has achieved remarkable success over the past couple of decades, often attributed to a combination of algorithmic innovations and the availability of high-quality data available at scale. However, a third critical component is the fine-tuning of hyperparameters, which plays a pivotal role in achieving optimal model performance. Despite its significance, hyperparameter optimization (HPO) remains a challenging task for several reasons. Many HPO techniques rely on naive search methods or assume that the loss function is smooth and continuous, which may not always be the case. Traditional methods, like grid search and Bayesian optimization, often struggle to quickly adapt and efficiently search the loss landscape. Grid search is computationally expensive, while Bayesian optimization can be slow to prime. Since the search space for HPO is frequently high-dimensional and non-convex, it is often challenging to efficiently find a global minimum. Moreover, optimal hyperparameters can be sensitive to the specific dataset or task, further complicating the search process. To address these issues, we propose a new hyperparameter optimization method, HomOpt, using a data-driven approach based on a generalized additive model (GAM) surrogate combined with homotopy optimization. This strategy augments established optimization methodologies to boost the performance and effectiveness of any given method with faster convergence to the optimum on continuous, discrete, and categorical domain spaces. We compare the effectiveness of HomOpt applied to multiple optimization techniques (e.g., Random Search, TPE, Bayes, and SMAC) showing improved objective performance on many standardized machine learning benchmarks and challenging open-set recognition tasks.

翻译：机器学习在过去几十年中取得了显著成功，这通常归因于算法创新与大规模高质量数据可用性的结合。然而，第三个关键因素——超参数的精细调整——在实现最优模型性能中起着关键作用。尽管超参数优化（HPO）至关重要，但由于多重原因，它仍是一项具有挑战性的任务。许多HPO技术依赖朴素搜索方法，或假设损失函数光滑连续，但实际情况可能并非如此。网格搜索和贝叶斯优化等传统方法往往难以快速适应并高效探索损失景观。网格搜索计算成本高昂，而贝叶斯优化启动缓慢。由于HPO的搜索空间通常是高维且非凸的，因此高效寻找全局最小值往往困难重重。此外，最优超参数可能对特定数据集或任务敏感，进一步增加了搜索过程的复杂性。为解决这些问题，我们提出了一种新的超参数优化方法HomOpt，该方法采用基于广义加性模型（GAM）代理与同伦优化相结合的数据驱动策略。该策略增强了既定优化方法，通过在连续、离散和分类域空间中更快收敛到最优值，提升任意方法的性能与有效性。我们将HomOpt应用于多种优化技术（如随机搜索、TPE、贝叶斯和SMAC）并比较其有效性，结果显示，在多个标准化机器学习基准测试和具有挑战性的开放集识别任务中，目标性能均得到了改善。