Hyperparameter tuning is critical to the success of federated learning applications. Unfortunately, appropriately selecting hyperparameters is challenging in federated networks. Issues of scale, privacy, and heterogeneity introduce noise in the tuning process and make it difficult to evaluate the performance of various hyperparameters. In this work, we perform the first systematic study on the effect of noisy evaluation in federated hyperparameter tuning. We first identify and rigorously explore key sources of noise, including client subsampling, data and systems heterogeneity, and data privacy. Surprisingly, our results indicate that even small amounts of noise can significantly impact tuning methods-reducing the performance of state-of-the-art approaches to that of naive baselines. To address noisy evaluation in such scenarios, we propose a simple and effective approach that leverages public proxy data to boost the evaluation signal. Our work establishes general challenges, baselines, and best practices for future work in federated hyperparameter tuning.

翻译：超参数调优对联邦学习应用的成功至关重要。然而，在联邦网络中恰当地选择超参数极具挑战性。规模、隐私和异构性问题在调优过程中引入噪声，使得评估不同超参数的性能变得困难。本文首次系统研究了联邦超参数调优中噪声评估的影响。我们首先识别并严格探讨了主要噪声来源，包括客户端子采样、数据与系统异构性及数据隐私。令人惊讶的是，我们的结果表明，即使少量噪声也能显著影响调优方法——将先进方法的性能降至朴素基线的水平。针对此类场景中的噪声评估问题，我们提出了一种简单有效的方法，利用公开代理数据增强评估信号。本工作为联邦超参数调优的未来研究确立了通用挑战、基线方法和最佳实践准则。