One of the most critical problems in machine learning is HyperParameter Optimization (HPO), since choice of hyperparameters has a significant impact on final model performance. Although there are many HPO algorithms, they either have no theoretical guarantees or require strong assumptions. To this end, we introduce BLiE -- a Lipschitz-bandit-based algorithm for HPO that only assumes Lipschitz continuity of the objective function. BLiE exploits the landscape of the objective function to adaptively search over the hyperparameter space. Theoretically, we show that $(i)$ BLiE finds an $\epsilon$-optimal hyperparameter with $O \left( \frac{1}{\epsilon} \right)^{d_z + \beta}$ total budgets, where $d_z$ and $\beta$ are problem intrinsic; $(ii)$ BLiE is highly parallelizable. Empirically, we demonstrate that BLiE outperforms the state-of-the-art HPO algorithms on benchmark tasks. We also apply BLiE to search for noise schedule of diffusion models. Comparison with the default schedule shows that BLiE schedule greatly improves the sampling speed.

翻译：机器学习中最重要的问题之一是超光谱优化(HPO),因为选择超光谱参数对最终模型性能有重大影响。虽然有许多高光谱算法,但它们要么没有理论保证,要么需要强有力的假设。为此,我们为高光谱算法引入BLiE -- -- 以利普西茨为基点的以利普西茨为基点的算法,它只假定目标功能的连续性。BLiE利用目标功能的景观,在超光谱空间上进行适应性搜索。理论上,我们显示BLE发现美元(一)优美的超光度参数,其值为$(left)(\frac{1-unsilon}\right) 或需要强有力的假设。为此,我们引入了BLE-z +\beta}总预算,它只假定利普西茨连续性,而美元和美元是内在问题;$(二)美元BLE是可高度平行的。从理论上看,我们证明BLE优于HPE的状态,它超越了HPo-art 扩散模型的状态,而最佳最佳超光谱超光谱,我们也应用了BPoxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx比比比比比比比比比比比比比比比比比比比比标准,我们也显示了BLLx的BLx的BBLxxxxxxxxxxxxxxxxxxxxx程比比。