One fascinating aspect of pre-trained vision-language models~(VLMs) learning under language supervision is their impressive zero-shot generalization capability. However, this ability is hindered by distribution shifts between the training and testing data. Previous test time adaptation~(TTA) methods for VLMs in zero-shot classification rely on minimizing the entropy of model outputs, tending to be stuck in incorrect model predictions. In this work, we propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. Specifically, a CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. Given a single test sample, the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution. The proposed \textit{reinforcement learning with CLIP feedback~(RLCF)} framework is highly flexible and universal. Beyond the classification task, with task-specific sampling strategies and a proper reward baseline choice, RLCF can be easily extended to not only discrimination tasks like retrieval but also generalization tasks like image captioning, improving the zero-shot generalization capacity of VLMs. According to the characteristics of these VL tasks, we build different fully TTA pipelines with RLCF to improve the zero-shot generalization ability of various VLMs. Extensive experiments along with promising empirical results demonstrate the effectiveness of RLCF. The code is available at https://github.com/mzhaoshuai/RLCF.

翻译：预训练视觉-语言模型在语言监督下学习的一个引人注目的特性是其惊人的零样本泛化能力。然而，这种能力受到训练数据与测试数据之间分布偏移的阻碍。先前针对视觉-语言模型零样本分类的测试时自适应方法依赖于最小化模型输出的熵，这容易使模型陷入错误的预测中。本文提出了一种带有反馈的测试时自适应方法，用于纠正模型输出并防止模型变得盲目自信。具体而言，在测试时自适应过程中，采用CLIP模型作为奖励模型，为视觉-语言模型提供反馈。对于单个测试样本，视觉-语言模型被强制最大化输入与从模型输出分布中采样结果之间的CLIP奖励。所提出的“基于CLIP反馈的强化学习”(RLCF)框架具有高度的灵活性和通用性。除分类任务外，通过设计任务特定的采样策略并选择合适的奖励基线，RLCF不仅可以轻松扩展到检索等判别任务，还可以扩展到图像描述等生成任务，从而提升视觉-语言模型的零样本泛化能力。针对这些视觉-语言任务的特点，我们利用RLCF构建了不同的完整测试时自适应流水线，以提升各类视觉-语言模型的零样本泛化能力。大量实验和令人鼓舞的实验结果证明了RLCF的有效性。代码可在https://github.com/mzhaoshuai/RLCF获取。