This work focuses on the problem of hyper-parameter tuning (HPT) for robust (i.e., adversarially trained) models, shedding light on the new challenges and opportunities arising during the HPT process for robust models. To this end, we conduct an extensive experimental study based on 3 popular deep models, in which we explore exhaustively 9 (discretized) HPs, 2 fidelity dimensions, and 2 attack bounds, for a total of 19208 configurations (corresponding to 50 thousand GPU hours). Through this study, we show that the complexity of the HPT problem is further exacerbated in adversarial settings due to the need to independently tune the HPs used during standard and adversarial training: succeeding in doing so (i.e., adopting different HP settings in both phases) can lead to a reduction of up to 80% and 43% of the error for clean and adversarial inputs, respectively. On the other hand, we also identify new opportunities to reduce the cost of HPT for robust models. Specifically, we propose to leverage cheap adversarial training methods to obtain inexpensive, yet highly correlated, estimations of the quality achievable using state-of-the-art methods. We show that, by exploiting this novel idea in conjunction with a recent multi-fidelity optimizer (taKG), the efficiency of the HPT process can be enhanced by up to 2.1x.

翻译：本文关注鲁棒（即对抗训练）模型的超参数调优（HPT）问题，揭示了鲁棒模型HPT过程中出现的新挑战与机遇。为此，我们基于3种主流深度模型开展了广泛的实验研究，全面探索了9个（离散化）超参数、2个保真度维度及2个攻击界，共计19208种配置（对应5万GPU小时）。通过该研究，我们表明：在对抗场景下，由于需要独立调整标准训练与对抗训练中使用的超参数，HPT问题的复杂性进一步加剧——若在此两方面成功采用不同的超参数设置，可使干净输入与对抗输入的误差分别降低高达80%和43%。另一方面，我们也识别出降低鲁棒模型HPT成本的新机遇，具体而言，提出利用廉价的对抗训练方法，获得与最优方法可达到的质量高度相关的低成本估计。实验表明，通过将此新颖思路与近期提出的多保真度优化器（taKG）相结合，HPT过程的效率可提升至2.1倍。