Large-scale vision-language models (LVLMs) pretrained on massive image-text pairs have achieved remarkable success in visual representations. However, existing paradigms to transfer LVLMs to downstream tasks encounter two primary challenges. Firstly, the text features remain fixed after being calculated and cannot be adjusted according to image features, which decreases the model's adaptability. Secondly, the model's output solely depends on the similarity between the text and image features, leading to excessive reliance on LVLMs. To address these two challenges, we introduce a novel two-branch model named the Instance-Wise Adaptive Tuning and Caching (ATC). Specifically, one branch implements our proposed ConditionNet, which guides image features to form an adaptive textual cache that adjusts based on image features, achieving instance-wise inference and improving the model's adaptability. The other branch introduces the similarities between images and incorporates a learnable visual cache, designed to decouple new and previous knowledge, allowing the model to acquire new knowledge while preserving prior knowledge. The model's output is jointly determined by the two branches, thus overcoming the limitations of existing methods that rely solely on LVLMs. Additionally, our method requires limited computing resources to tune parameters, yet outperforms existing methods on 11 benchmark datasets.

翻译：大规模视觉语言模型（LVLM）在海量图像-文本对上的预训练已在视觉表征方面取得了显著成功。然而，现有将LVLM迁移至下游任务的范式面临两大挑战：首先，文本特征在计算后保持固定，无法根据图像特征进行调整，降低了模型的自适应能力；其次，模型输出仅依赖于文本与图像特征的相似度，导致对LVLM的过度依赖。为解决这两个问题，我们提出了一种新颖的双分支模型——实例级自适应调优与缓存（ATC）。具体而言，一个分支实现了我们提出的ConditionNet，它引导图像特征形成基于图像特征动态调整的自适应文本缓存，从而实现实例级推理并提升模型适应性；另一分支引入图像间的相似度并集成可学习的视觉缓存，旨在解耦新旧知识，使模型在保留先验知识的同时获取新知识。模型的输出由两个分支共同决定，从而克服了现有方法仅依赖LVLM的局限性。此外，我们的方法仅需有限的计算资源即可进行参数调优，但在11个基准数据集上的性能优于现有方法。