Foundation models encode rich representations that can be adapted to a desired task by fine-tuning on task-specific data. However, fine-tuning a model on one particular data distribution often compromises the model's original performance on other distributions. Current methods for robust fine-tuning utilize hand-crafted regularization techniques to constrain the fine-tuning process towards the base foundation model. Yet, it is hard to precisely specify what characteristics of the foundation model to retain during fine-tuning, as this depends on how the pre-training, fine-tuning, and evaluation data distributions relate to each other. We propose AutoFT, a data-driven approach for guiding foundation model fine-tuning. AutoFT optimizes fine-tuning hyperparameters to maximize performance on a small out-of-distribution (OOD) validation set. To guide fine-tuning in a granular way, AutoFT searches a highly expressive hyperparameter space that includes weight coefficients for many different losses, in addition to learning rate and weight decay values. We evaluate AutoFT on nine natural distribution shifts which include domain shifts and subpopulation shifts. Our experiments show that AutoFT significantly improves generalization to new OOD data, outperforming existing robust fine-tuning methods. Notably, AutoFT achieves new state-of-the-art performance on the WILDS-iWildCam and WILDS-FMoW benchmarks, outperforming the previous best methods by $6.0\%$ and $1.5\%$, respectively.

翻译：基础模型编码了丰富的表征，可通过在特定任务数据上微调来适配目标任务。然而，针对某一特定数据分布微调模型往往会损害模型在其他分布上的原始性能。当前鲁棒微调方法采用手工设计的正则化技术，将微调过程约束在基础模型附近。但由于预训练、微调和评估数据分布之间的关联方式难以精确界定，因此很难明确指定微调过程中应保留基础模型的哪些特征。我们提出AutoFT——一种数据驱动的引导基础模型微调方法。AutoFT通过优化微调超参数，在小型分布外（OOD）验证集上最大化性能。为实现精细化的微调引导，AutoFT在包含学习率与权重衰减值的基础上，进一步搜索了涵盖多种损失权重的超高表达性超参数空间。我们在九种自然分布偏移（包括领域偏移和子群体偏移）上评估了AutoFT。实验表明，AutoFT显著提升了新OOD数据的泛化能力，性能超越现有鲁棒微调方法。值得注意的是，AutoFT在WILDS-iWildCam和WILDS-FMoW基准测试中达到了新的最优性能，分别比此前最佳方法提升$6.0\%$和$1.5\%$。