Hyperparameter optimization (HPO) is crucial for fine-tuning machine learning models but can be computationally expensive. To reduce costs, Multi-fidelity HPO (MF-HPO) leverages intermediate accuracy levels in the learning process and discards low-performing models early on. We compared various representative MF-HPO methods against a simple baseline on classical benchmark data. The baseline involved discarding all models except the Top-K after training for only one epoch, followed by further training to select the best model. Surprisingly, this baseline achieved similar results to its counterparts, while requiring an order of magnitude less computation. Upon analyzing the learning curves of the benchmark data, we observed a few dominant learning curves, which explained the success of our baseline. This suggests that researchers should (1) always use the suggested baseline in benchmarks and (2) broaden the diversity of MF-HPO benchmarks to include more complex cases.

翻译：超参数优化（HPO）对于微调机器学习模型至关重要，但计算成本高昂。为降低成本，多保真度HPO（MF-HPO）利用学习过程中的中间精度级别，并提前淘汰性能较差的模型。我们在经典基准数据上将多种代表性MF-HPO方法与一个简单基线进行了比较。该基线仅训练一轮后便剔除除Top-K之外的所有模型，再通过进一步训练筛选出最佳模型。令人惊讶的是，该基线在计算量降低一个数量级的情况下，取得了与对比方法相似的结果。通过分析基准数据的学习曲线，我们观察到少数主导学习曲线的存在，这解释了基线的成功。这表明研究者应当：（1）在基准测试中始终使用建议的基线；（2）扩大MF-HPO基准测试的多样性，纳入更复杂的案例。