Hyperparameter (HP) optimization of deep learning (DL) is essential for high performance. As DL often requires several hours to days for its training, HP optimization (HPO) of DL is often prohibitively expensive. This boosted the emergence of tabular or surrogate benchmarks, which enable querying the (predictive) performance of DL with a specific HP configuration in a fraction. However, since the actual runtime of a DL training is significantly different from its query response time, simulators of an asynchronous HPO, e.g. multi-fidelity optimization, must wait for the actual runtime at each iteration in a na\"ive implementation; otherwise, the evaluation order during simulation does not match with the real experiment. To ease this issue, we developed a Python wrapper and describe its usage. This wrapper forces each worker to wait so that we yield exactly the same evaluation order as in the real experiment with only $10^{-2}$ seconds of waiting instead of waiting several hours. Our implementation is available at https://github.com/nabenabe0928/mfhpo-simulator/.

翻译：深度学习（DL）的超参数（HP）优化对于实现高性能至关重要。由于DL的训练通常需要数小时乃至数天，其超参数优化（HPO）往往代价高昂。这一现状推动了表格型或替代性基准的出现，这些基准能够以极短时间查询特定HP配置下DL的（预测性）性能。然而，由于DL训练的实际运行时间与查询响应时间存在显著差异，异步HPO（如多保真优化）的模拟器在朴素实现中必须等待每次迭代的实际运行时间；否则模拟过程中的评估顺序将与真实实验不一致。为解决此问题，我们开发了一个Python包装器并描述其使用方法。该包装器强制每个工作节点等待，从而在仅需$10^{-2}$秒等待（而非数小时）的情况下，实现与真实实验完全相同的评估顺序。我们的实现代码已开源至https://github.com/nabenabe0928/mfhpo-simulator/。