Hyperparameter tuning is a common practice in the application of machine learning but is a typically ignored aspect in the literature on privacy-preserving machine learning due to its negative effect on the overall privacy parameter. In this paper, we aim to tackle this fundamental yet challenging problem by providing an effective hyperparameter tuning framework with differential privacy. The proposed method allows us to adopt a broader hyperparameter search space and even to perform a grid search over the whole space, since its privacy loss parameter is independent of the number of hyperparameter candidates. Interestingly, it instead correlates with the utility gained from hyperparameter searching, revealing an explicit and mandatory trade-off between privacy and utility. Theoretically, we show that its additional privacy loss bound incurred by hyperparameter tuning is upper-bounded by the squared root of the gained utility. However, we note that the additional privacy loss bound would empirically scale like a squared root of the logarithm of the utility term, benefiting from the design of doubling step.

翻译：超参数调优是机器学习应用中的常见实践，但在隐私保护机器学习文献中却因对整体隐私参数产生负面影响而通常被忽视。本文旨在通过提供一种具有差分隐私有效性的超参数调优框架，来解决这一基础且具有挑战性的问题。所提方法使我们能够采用更广泛的超参数搜索空间，甚至在整个空间执行网格搜索，因为其隐私损失参数与超参数候选数量无关。有趣的是，该参数反而与超参数搜索所获得的效用相关，揭示了隐私与效用之间明确且强制性的权衡。理论上，我们证明由超参数调优引发的额外隐私损失边界受限于所获效用的平方根。然而，我们注意到得益于倍增步骤的设计，该额外隐私损失边界在实际中会按效用项对数的平方根尺度扩展。