Conventional wisdom suggests parameter-efficient fine-tuning of foundation models as the state-of-the-art method for transfer learning in vision, replacing the rich literature of alternatives such as meta-learning. In trying to harness the best of both worlds, meta-tuning introduces a subsequent optimization stage of foundation models but has so far only shown limited success and crucially tends to underperform on out-of-domain (OOD) tasks. In this paper, we introduce Sparse MetA-Tuning (SMAT), a method inspired by sparse mixture-of-experts approaches and trained to isolate subsets of pre-trained parameters automatically for meta-tuning on each task. SMAT successfully overcomes OOD sensitivity and delivers on the promise of enhancing the transfer abilities of vision foundation models beyond parameter-efficient finetuning. We establish new state-of-the-art results on a challenging combination of Meta-Dataset augmented with additional OOD tasks in both zero-shot and gradient-based adaptation settings. In addition, we provide a thorough analysis of the superiority of learned over hand-designed sparsity patterns for sparse expert methods and the pivotal importance of the sparsity level in balancing between in-domain and out-of-domain generalization. Our code is publicly available.

翻译：传统观点认为，对基础模型进行参数高效微调是视觉迁移学习的最先进方法，取代了元学习等丰富的替代方案。为了融合两者优势，元调优引入了基础模型的后续优化阶段，但迄今为止仅展现出有限成功，且关键是在跨领域（OOD）任务上往往表现不佳。本文提出稀疏元调优（SMAT），该方法受稀疏混合专家方法启发，训练过程中自动为每个任务隔离预训练参数的子集。SMAT成功克服了对OOD的敏感性，实现了超越参数高效微调来增强视觉基础模型迁移能力的承诺。我们在包含额外OOD任务的挑战性元数据集组合上，在零样本和基于梯度的自适应设置下均取得了新的最先进结果。此外，我们深入分析了学习型稀疏模式相较于手工设计模式在稀疏专家方法中的优越性，以及稀疏度在平衡领域内与跨领域泛化中的关键作用。我们的代码已公开。