Recently, foundation models based on Vision Transformers (ViTs) have become widely available. However, their fine-tuning process is highly resource-intensive, and it hinders their adoption in several edge or low-energy applications. To this end, in this paper we introduce an efficient fine-tuning method for ViTs called $\textbf{ALaST}$ ($\textit{Adaptive Layer Selection Fine-Tuning for Vision Transformers}$) to speed up the fine-tuning process while reducing computational cost, memory load, and training time. Our approach is based on the observation that not all layers are equally critical during fine-tuning, and their importance varies depending on the current mini-batch. Therefore, at each fine-tuning step, we adaptively estimate the importance of all layers and we assign what we call ``compute budgets'' accordingly. Layers that were allocated lower budgets are either trained with a reduced number of input tokens or kept frozen. Freezing a layer reduces the computational cost and memory usage by preventing updates to its weights, while discarding tokens removes redundant data, speeding up processing and reducing memory requirements. We show that this adaptive compute allocation enables a nearly-optimal schedule for distributing computational resources across layers, resulting in substantial reductions in training time (up to 1.5x), FLOPs (up to 2x), and memory load (up to 2x) compared to traditional full fine-tuning approaches. Additionally, it can be successfully combined with other parameter-efficient fine-tuning methods, such as LoRA.

翻译：近年来，基于视觉Transformer（ViT）的基础模型已得到广泛应用。然而，其微调过程对计算资源需求极高，这阻碍了它们在多种边缘或低功耗场景中的部署。为此，本文提出一种高效的ViT微调方法，称为$\textbf{ALaST}$（$\textit{Adaptive Layer Selection Fine-Tuning for Vision Transformers}$），旨在加速微调过程，同时降低计算开销、内存负载与训练时间。我们的方法基于以下观察：在微调过程中，并非所有层都同等关键，且各层的重要性随当前小批次数据的变化而不同。因此，在每一微调步骤中，我们自适应地估计所有层的重要性，并据此分配所谓的“计算预算”。被分配较低预算的层，其训练过程将采用减少的输入令牌数量，或保持冻结状态。冻结层可避免权重更新，从而降低计算成本与内存占用；而丢弃令牌则能移除冗余数据，加速处理并减少内存需求。实验表明，这种自适应计算分配机制能够实现近乎最优的跨层计算资源调度方案，与传统全参数微调方法相比，可显著减少训练时间（最高达1.5倍）、浮点运算量（最高达2倍）及内存负载（最高达2倍）。此外，该方法可成功与其他参数高效微调方法（如LoRA）结合使用。