Out-of-distribution (OOD) generalization, where the model needs to handle distribution shifts from training, is a major challenge of machine learning. Recently, contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, revealing a promising path toward OOD generalization. However, to boost upon zero-shot performance, further adaptation of CLIP on downstream tasks is indispensable but undesirably degrades OOD generalization ability. In this paper, we aim at generalizing CLIP to out-of-distribution test data on downstream tasks. Beyond the two canonical OOD situations, domain shift and open class, we tackle a more general but difficult in-the-wild setting where both OOD situations may occur on the unseen test data. We propose CLIPood, a simple fine-tuning method that can adapt CLIP models to all OOD situations. To exploit semantic relations between classes from the text modality, CLIPood introduces a new training objective, margin metric softmax (MMS), with class adaptive margins for fine-tuning. Moreover, to incorporate both the pre-trained zero-shot model and the fine-tuned task-adaptive model, CLIPood proposes a new Beta moving average (BMA) to maintain a temporal ensemble according to Beta distribution. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.

翻译：分布外（OOD）泛化——模型需处理训练数据分布偏移的问题——是机器学习的主要挑战。近期，对比语言-图像预训练（CLIP）模型展现出惊人的零样本能力，为OOD泛化开辟了有前景的路径。然而，为提升零样本性能，对CLIP在下游任务中进行进一步适配必不可少，但这会非期望地削弱其OOD泛化能力。本文旨在将CLIP泛化至下游任务的分布外测试数据上。除领域偏移和开放类别两类经典OOD场景外，我们应对更普遍但困难的"野外"场景——即未观测测试数据中可能同时出现上述两种OOD情形。我们提出CLIPood，一种能适配CLIP模型至所有OOD场景的简洁微调方法。为利用文本模态中类别间的语义关联，CLIPood引入新型训练目标——带类别自适应边界的边际度量Softmax（MMS）进行微调。此外，为融合预训练零样本模型与微调后的任务自适应模型，CLIPood提出新型Beta移动平均（BMA）方法，通过Beta分布维护时序集成。在包含不同OOD场景的多类数据集上的实验表明，CLIPood性能始终优于现有泛化技术。