The Vision-Language Pre-training (VLP) models like CLIP have gained popularity in recent years. However, many works found that the social biases hidden in CLIP easily manifest in downstream tasks, especially in image retrieval, which can have harmful effects on human society. In this work, we propose FairCLIP to eliminate the social bias in CLIP-based image retrieval without damaging the retrieval performance achieving the compatibility between the debiasing effect and the retrieval performance. FairCLIP is divided into two steps: Attribute Prototype Learning (APL) and Representation Neutralization (RN). In the first step, we extract the concepts needed for debiasing in CLIP. We use the query with learnable word vector prefixes as the extraction structure. In the second step, we first divide the attributes into target and bias attributes. By analysis, we find that both attributes have an impact on the bias. Therefore, we try to eliminate the bias by using Re-Representation Matrix (RRM) to achieve the neutralization of the representation. We compare the debiasing effect and retrieval performance with other methods, and experiments demonstrate that FairCLIP can achieve the best compatibility. Although FairCLIP is used to eliminate bias in image retrieval, it achieves the neutralization of the representation which is common to all CLIP downstream tasks. This means that FairCLIP can be applied as a general debiasing method for other fairness issues related to CLIP.

翻译：近年来，视觉语言预训练模型（如CLIP）日益普及。然而，许多研究发现，CLIP中隐含的社会偏见极易在下游任务中显现，尤其在图像检索领域，这可能对人类社产生有害影响。本文提出FairCLIP方法，旨在消除基于CLIP的图像检索中的社会偏见，同时保持检索性能，实现去偏效果与检索性能的兼容性。FairCLIP分为两个步骤：属性原型学习与表征中立化。第一步，我们提取CLIP中用于去偏的概念，采用可学习词向量前缀的查询作为提取结构。第二步，我们首先将属性划分为目标属性与偏见属性。通过分析发现，两类属性均对偏见产生影响。因此，我们尝试通过重表征矩阵实现表征中立化以消除偏见。通过与其他方法在去偏效果和检索性能方面的对比实验表明，FairCLIP能够实现最佳的兼容性。尽管FairCLIP用于消除图像检索中的偏见，但其实现的表征中立化适用于所有CLIP下游任务。这意味着FairCLIP可作为一种通用去偏方法，应用于其他与CLIP相关的公平性问题。