Despite the growing demand for tuning foundation vision transformers (FViTs) on downstream tasks, fully unleashing FViTs' potential under data-limited scenarios (e.g., few-shot tuning) remains a challenge due to FViTs' data-hungry nature. Common data augmentation techniques fall short in this context due to the limited features contained in the few-shot tuning data. To tackle this challenge, we first identify an opportunity for FViTs in few-shot tuning: pretrained FViTs themselves have already learned highly representative features from large-scale pretraining data, which are fully preserved during widely used parameter-efficient tuning. We thus hypothesize that leveraging those learned features to augment the tuning data can boost the effectiveness of few-shot FViT tuning. To this end, we propose a framework called Hint-based Data Augmentation (Hint-Aug), which aims to boost FViT in few-shot tuning by augmenting the over-fitted parts of tuning samples with the learned features of pretrained FViTs. Specifically, Hint-Aug integrates two key enablers: (1) an Attentive Over-fitting Detector (AOD) to detect over-confident patches of foundation ViTs for potentially alleviating their over-fitting on the few-shot tuning data and (2) a Confusion-based Feature Infusion (CFI) module to infuse easy-to-confuse features from the pretrained FViTs with the over-confident patches detected by the above AOD in order to enhance the feature diversity during tuning. Extensive experiments and ablation studies on five datasets and three parameter-efficient tuning techniques consistently validate Hint-Aug's effectiveness: 0.04% ~ 32.91% higher accuracy over the state-of-the-art (SOTA) data augmentation method under various low-shot settings. For example, on the Pet dataset, Hint-Aug achieves a 2.22% higher accuracy with 50% less training data over SOTA data augmentation methods.

翻译：尽管对下游任务进行基础视觉Transformer（FViTs）调优的需求日益增长，但由于FViTs对数据的高依赖性，在数据受限场景下充分释放其潜力仍具挑战。传统数据增强技术在此场景下效果有限，因为少样本调优数据所含特征较为贫乏。为应对这一挑战，我们首先发现FViTs在少样本调优中的机遇：预训练的FViTs已从大规模预训练数据中学习到高度表征性特征，这些特征在广泛使用的参数高效微调中得以完整保留。因此，我们假设利用这些已学特征增强调优数据，可提升FViTs少样本调优的有效性。为此，我们提出基于提示的数据增强框架（Hint-Aug），旨在通过利用预训练FViTs的已学特征增强调优样本的过拟合部分，进而提升FViTs少样本调优效果。具体而言，Hint-Aug集成两个关键模块：（1）注意力过拟合检测器（AOD），用于检测基础ViTs的过度置信补丁，以缓解其在少样本调优数据上的过拟合；（2）基于混淆的特征注入（CFI）模块，将预训练FViTs中的易混淆特征注入上述AOD检测到的过度置信补丁，以增强调优过程中的特征多样性。在五个数据集和三种参数高效调优技术上进行的广泛实验与消融研究一致验证了Hint-Aug的有效性：在各类少样本设置下，相比最先进（SOTA）数据增强方法，准确率提升0.04%至32.91%。例如，在Pet数据集上，Hint-Aug使用减少50%的训练数据，仍比SOTA数据增强方法实现2.22%的准确率提升。