This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts (Fig. \ref{fig:fig1} a). Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance (Fig. \ref{fig:fig1} b). Our method achieves a 12.41\% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing. Comprehensive experiments conducted on two widely-used datasets, encompassing 20 generation models, validate the efficacy of the proposed method, demonstrating state-of-the-art performance. The code is available at \url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}
翻译:本研究聚焦于AIGC检测,旨在开发能够识别各类伪造图像的通用检测器。近期研究发现,结合线性分类器,CLIP等大型预训练模型对于可泛化的深度伪造检测具有显著效果。然而,两个关键问题仍未解决:1)理解为何CLIP特征通过线性分类器能在深度伪造检测中表现优异;2)探索CLIP的检测潜力。本研究通过将CLIP的检测特征解码为文本并进行词频分析,深入探究了其检测能力的内在机制。我们的发现表明,CLIP通过识别相似概念来检测深度伪造内容(图\ref{fig:fig1} a)。基于此洞见,我们提出了类别通用提示CLIP(简称C2P-CLIP),该方法将类别通用提示集成到文本编码器中,从而向图像编码器注入与类别相关的概念,以提升检测性能(图\ref{fig:fig1} b)。相较于原始CLIP,我们的方法在测试阶段未引入额外参数的情况下,实现了12.41%的检测精度提升。在两个广泛使用的数据集(涵盖20种生成模型)上进行的全面实验验证了所提方法的有效性,展现了最先进的性能。代码发布于\url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}