Adjustable hyperparameters of machine learning models typically impact various key trade-offs such as accuracy, fairness, robustness, or inference cost. Our goal in this paper is to find a configuration that adheres to user-specified limits on certain risks while being useful with respect to other conflicting metrics. We solve this by combining Bayesian Optimization (BO) with rigorous risk-controlling procedures, where our core idea is to steer BO towards an efficient testing strategy. Our BO method identifies a set of Pareto optimal configurations residing in a designated region of interest. The resulting candidates are statistically verified and the best-performing configuration is selected with guaranteed risk levels. We demonstrate the effectiveness of our approach on a range of tasks with multiple desiderata, including low error rates, equitable predictions, handling spurious correlations, managing rate and distortion in generative models, and reducing computational costs.

翻译：机器学习模型的可调超参数通常会影响多个关键权衡指标，如精度、公平性、鲁棒性或推理成本。本文旨在寻找一种配置，既能满足用户对特定风险的限定要求，又能在其他冲突指标上保持实用性。我们通过将贝叶斯优化（BO）与严格的风险控制流程相结合来解决此问题，其核心思想是引导BO采用高效的测试策略。我们的BO方法能够识别出位于指定感兴趣区域内的帕累托最优配置集。这些候选配置经过统计验证后，在保证风险水平的前提下选择性能最优的配置。我们在多个具有多重需求的场景中验证了该方法的效果，这些场景包括低错误率、公平预测、处理虚假相关性、管理生成模型中的码率与失真，以及降低计算成本。