Recent empirical work shows that inconsistent results, based on choice of hyperparameter optimization (HPO) configuration, are a widespread problem in ML research. When comparing two algorithms J and K, searching one subspace can yield the conclusion that J outperforms K, whereas searching another can entail the opposite. In short, the way we choose hyperparameters can deceive us. We provide a theoretical complement to this prior work, arguing that, to avoid such deception, the process of drawing conclusions from HPO should be made more rigorous. We call this process epistemic hyperparameter optimization (EHPO), and put forth a logical framework to capture its semantics and how it can lead to inconsistent conclusions about performance. Our framework enables us to prove EHPO methods that are guaranteed to be defended against deception. We demonstrate its utility by proving and empirically validating a defended variant of random search.

翻译：最近的经验工作表明,基于选择超参数优化(HPO)配置的不一致结果在ML研究中是一个普遍的问题。在比较两种算法J和K时,搜索一个子空间可以得出J优于K的结论,而搜索另一个子空间则可能产生相反的结果。简而言之,我们选择超参数的方式可以欺骗我们。我们为先前的这项工作提供了理论补充,认为为了避免这种欺骗,应当使从HPO得出结论的过程更加严格。我们称之为超参数优化(EHPO),并提出了一个逻辑框架来捕捉它的语义和如何导致关于性能的不一致的结论。我们的框架使我们能够证明EHPO方法有保证不受欺骗。我们通过证明和实验性地验证随机搜索的防御变体来证明它的效用。