The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.

翻译：基础模型（如分割一切模型SAM）的兴起，激发了参数高效微调（PEFT）方法的发展，这些方法将大型模型适配至其训练数据之外的应用领域。然而，不同PEFT技术对模型表征的修改方式各异，使得为特定领域选择最合适的方法成为一项具有挑战性的任务。受传统混合专家（MoE）方法的启发，我们提出了一种新框架——混合式PEFT方法（MoPEFT），并将其用于微调SAM。MoPEFT框架包含三种不同的PEFT子模块技术，能够动态学习并激活最适合特定数据任务配置的子模块。我们在分割一切模型上测试了该方法，结果表明MoPEFT在MESS基准测试中始终优于其他微调方法。