Foundational Vision-Language models such as CLIP have exhibited impressive generalization in downstream tasks. However, CLIP suffers from a two-level misalignment issue, i.e., task misalignment and data misalignment, when adapting to specific tasks. Soft prompt tuning has mitigated the task misalignment, yet the data misalignment remains a challenge. To analyze the impacts of the data misalignment, we revisit the pre-training and adaptation processes of CLIP and develop a structural causal model. We discover that while we expect to capture task-relevant information for downstream tasks accurately, the task-irrelevant knowledge impacts the prediction results and hampers the modeling of the true relationships between the images and the predicted classes. As task-irrelevant knowledge is unobservable, we leverage the front-door adjustment and propose Causality-Guided Semantic Decoupling and Classification (CDC) to mitigate the interference of task-irrelevant knowledge. Specifically, we decouple semantics contained in the data of downstream tasks and perform classification based on each semantic. Furthermore, we employ the Dempster-Shafer evidence theory to evaluate the uncertainty of each prediction generated by diverse semantics. Experiments conducted in multiple different settings have consistently demonstrated the effectiveness of CDC.

翻译：基础视觉语言模型（如CLIP）在下游任务中展现出卓越的泛化能力。然而，在适应特定任务时，CLIP存在双重错位问题，即任务错位与数据错位。软提示调优已缓解了任务错位，但数据错位仍是待解决的挑战。为分析数据错位的影响，我们重新审视CLIP的预训练与适应过程，构建了一个结构因果模型。研究发现，尽管我们期望准确捕捉下游任务相关的信息，但任务无关知识会影响预测结果，并阻碍图像与预测类别间真实关系的建模。由于任务无关知识不可观测，我们借助前门调整方法，提出因果引导的语义解耦与分类框架（CDC）以减轻任务无关知识的干扰。具体而言，我们对下游任务数据所包含的语义进行解耦，并基于各语义分别进行分类。此外，我们采用Dempster-Shafer证据理论评估不同语义生成预测的不确定性。在多种不同设置下进行的实验均一致证明了CDC的有效性。