The promising zero-shot generalization of vision-language models such as CLIP has led to their adoption using prompt learning for numerous downstream tasks. Previous works have shown test-time prompt tuning using entropy minimization to adapt text prompts for unseen domains. While effective, this overlooks the key cause for performance degradation to unseen domains -- distribution shift. In this work, we explicitly handle this problem by aligning the out-of-distribution (OOD) test sample statistics to those of the source data using prompt tuning. We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Evaluating against the domain generalization benchmark, our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe. In cross-dataset generalization with unseen categories across 10 datasets, our method improves consistently across all datasets compared to the existing state-of-the-art. Our source code and models are available at https://jameelhassan.github.io/promptalign.

翻译：视觉-语言模型（如CLIP）在零样本泛化方面展现出巨大潜力，促使人们通过提示学习将其应用于众多下游任务。以往研究利用熵最小化在测试时调整文本提示以适应未见过的领域，但该方法忽略了导致在未见过领域性能下降的关键原因——分布偏移。本文通过使用提示调整，将分布外测试样本的统计数据与源数据对齐，明确处理了该问题。我们利用单个测试样本在测试时调整多模态提示，通过最小化特征分布偏移来弥合测试领域中的差距。在领域泛化基准上的评估表明，我们的方法比现有提示学习技术提升了零样本Top-1准确率，相较于基线MaPLe提高了3.08%。在跨数据集泛化（涉及10个数据集的未见类别）中，我们的方法在所有数据集上均一致优于现有最先进方法。我们的源代码和模型已开源在https://jameelhassan.github.io/promptalign。