Generative models can create entirely new images, but they can also partially modify real images in ways that are undetectable to the human eye. In this paper, we address the challenge of automatically detecting such local manipulations. One of the most pressing problems in deepfake detection remains the ability of models to generalize to different classes of generators. In the case of fully manipulated images, representations extracted from large self-supervised models (such as CLIP) provide a promising direction towards more robust detectors. Here, we introduce DeCLIP, a first attempt to leverage such large pretrained features for detecting local manipulations. We show that, when combined with a reasonably large convolutional decoder, pretrained self-supervised representations are able to perform localization and improve generalization capabilities over existing methods. Unlike previous work, our approach is able to perform localization on the challenging case of latent diffusion models, where the entire image is affected by the fingerprint of the generator. Moreover, we observe that this type of data, which combines local semantic information with a global fingerprint, provides more stable generalization than other categories of generative methods.

翻译：生成模型能够创建全新的图像，同时也能以人眼无法察觉的方式部分修改真实图像。本文致力于解决自动检测此类局部篡改的挑战。深度伪造检测中最紧迫的问题之一仍然是模型对不同类别生成器的泛化能力。对于完全篡改的图像，从大型自监督模型（如CLIP）中提取的表征为构建更鲁棒的检测器提供了有前景的方向。在此，我们提出DeCLIP，这是首次尝试利用此类大规模预训练特征来检测局部篡改。研究表明，当与规模适中的卷积解码器结合时，预训练的自监督表征能够执行定位任务，并在泛化能力上超越现有方法。与先前工作不同，我们的方法能够在具有挑战性的潜在扩散模型案例中实现定位，此类模型中整个图像均受到生成器指纹的影响。此外，我们观察到这类同时包含局部语义信息与全局指纹的数据，相比其他类别的生成方法能提供更稳定的泛化性能。