With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset. In this paper, we propose a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pretrained models with multiple hyperparameter configurations on a series of datasets. To this aim, we evaluated over 20k hyperparameter configurations for finetuning 24 pretrained image classification models on 87 datasets to generate a large-scale meta-dataset. We meta-learn a multi-fidelity performance predictor on the learning curves of this meta-dataset and use it for fast hyperparameter optimization on new datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for a new dataset together with its optimal hyperparameters.

翻译：随着预测练模型数量不断增加，机器学习从业者持续面临如何选择预测练模型以及如何针对新数据集进行微调的问题。本文提出一种联合搜索最优预测练模型及其微调超参数的方法。该方法将多个预测练模型在不同超参数配置下在系列数据集上的性能知识进行迁移。为此，我们在87个数据集上评估了24个预测练图像分类模型的超过2万组超参数配置，生成了大规模元数据集。我们基于该元数据集的学习曲线元学习了一个多保真度性能预测器，并将其用于新数据集的快速超参数优化。实验证明，该方法能快速为新数据集选择准确的预测练模型及其最优超参数。