Hyperparameter optimization (HPO) is important to leverage the full potential of machine learning (ML). In practice, users are often interested in multi-objective (MO) problems, i.e., optimizing potentially conflicting objectives, like accuracy and energy consumption. To tackle this, the vast majority of MO-ML algorithms return a Pareto front of non-dominated machine learning models to the user. Optimizing the hyperparameters of such algorithms is non-trivial as evaluating a hyperparameter configuration entails evaluating the quality of the resulting Pareto front. In literature, there are known indicators that assess the quality of a Pareto front (e.g., hypervolume, R2) by quantifying different properties (e.g., volume, proximity to a reference point). However, choosing the indicator that leads to the desired Pareto front might be a hard task for a user. In this paper, we propose a human-centered interactive HPO approach tailored towards multi-objective ML leveraging preference learning to extract desiderata from users that guide the optimization. Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator. Concretely, we leverage pairwise comparisons of distinct Pareto fronts to learn such an appropriate quality indicator. Then, we optimize the hyperparameters of the underlying MO-ML algorithm towards this learned indicator using a state-of-the-art HPO approach. In an experimental study targeting the environmental impact of ML, we demonstrate that our approach leads to substantially better Pareto fronts compared to optimizing based on a wrong indicator pre-selected by the user, and performs comparable in the case of an advanced user knowing which indicator to pick.

翻译：超参数优化（HPO）对于充分发挥机器学习（ML）的潜力至关重要。在实践中，用户通常关注多目标（MO）问题，即优化可能存在冲突的目标（如准确率和能耗）。为解决此问题，绝大多数多目标机器学习算法会向用户返回一个由非支配机器学习模型构成的帕累托前沿。优化此类算法的超参数并非易事，因为评估一个超参数配置需要评估所生成的帕累托前沿的质量。文献中已有多种已知指标（如超体积、R2）通过量化不同属性（如体积、与参考点的接近度）来评估帕累托前沿的质量。然而，对于用户而言，选择能够生成期望帕累托前沿的指标可能是一项艰巨任务。本文提出一种以人为中心、面向多目标机器学习的交互式超参数优化方法，该方法利用偏好学习从用户中提取引导优化的需求。我们的方法不依赖用户猜测最适合其需求的指标，而是自动学习一个合适的指标。具体而言，我们通过不同帕累托前沿的两两比较来学习这种恰当的质量指标。随后，我们采用最先进的HPO方法，基于该学习到的指标优化底层MO-ML算法的超参数。在一项针对机器学习环境影响的实验研究中，我们证明：与用户预先错误选择指标进行优化相比，我们的方法能够生成明显更优的帕累托前沿；而在用户具备高阶知识、能够准确选择指标的情况下，其性能与该方法相当。