Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness.

翻译：超参数优化构成了典型现代机器学习工作流程的重要组成部分。这源于机器学习方法及相应的预处理步骤通常只有在超参数得到适当调优时才能获得最佳性能。但在许多应用中，我们不仅关注优化机器学习流程的预测准确性；在确定最优配置时还需考虑额外指标或约束条件，从而形成多目标优化问题。由于缺乏多目标超参数优化的相关知识和现成的软件实现，该问题在实践中常被忽视。本文向读者介绍多目标超参数优化的基本原理，并阐述其在应用机器学习中的实用价值。此外，我们对现有优化策略进行了全面综述，涵盖进化算法和贝叶斯优化两大领域。我们通过多个具体机器学习应用场景展示多目标优化的实用性，涉及运行条件、预测时间、稀疏性、公平性、可解释性和鲁棒性等优化目标。