Fine-tuning pre-trained foundation models has gained significant popularity in various research fields. Existing methods for fine-tuning can be roughly divided into two categories, namely Parameter-Efficient Fine-Tuning and High-Performance Fine-Tuning. The former aims at improving efficiency, while the latter focuses on enhancing performance. Beyond these methods, we demonstrate that Partial Fine-Tuning can be an innovative and promising direction capable of concurrently enhancing both efficiency and accuracy. We first validate eight manually-defined partial fine-tuning strategies across kinds of datasets and vision transformer architectures, and find that some partial fine-tuning strategies (e.g., ffn only or attention only) can achieve better performance with fewer tuned parameters than full fine-tuning, and selecting appropriate layers is critical to partial fine-tuning. Thus, we propose a novel fine-tuned angle metric to guide the selection of appropriate layers for partial fine-tuning, making it flexible to be adapted to various scenarios for more practicable partial fine-tuning. Additionally, we show that partial fine-tuning can serve as a new dimension for Model Soups, improving both the model performance and generalization with fewer tuned parameters. Comprehensive experiments on a wide range of datasets and models validate the great potential of partial fine-tuning.

翻译：微调预训练基础模型已在各个研究领域获得广泛关注。现有微调方法大致可分为两类：参数高效微调与高性能微调。前者旨在提升效率，后者则侧重于优化性能。除这些方法外，我们证明部分微调可成为一种兼具效率与准确性的创新性且前景广阔的方向。我们首先在多种数据集和视觉Transformer架构上验证了八种人工定义的部分微调策略，发现部分策略（如仅微调前馈网络或仅微调注意力模块）能以更少的可调参数实现优于全微调的性能，且选择适当的网络层对部分微调至关重要。为此，我们提出一种新型的微调角度度量指标，用于指导部分微调中合适网络层的选取，使其能灵活适应各类场景以实现更实用的部分微调。此外，我们证明部分微调可作为模型汤（Model Soups）的新维度，以更少的可调参数同时提升模型性能与泛化能力。在广泛数据集和模型上的综合实验验证了部分微调的巨大潜力。