Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models -- which we refer to as d\'ej\`a vu memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that d\'ej\`a vu memorization is common to different SSL algorithms, is exacerbated by certain design choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of d\'ej\`a vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies. Code is available at https://github.com/facebookresearch/DejaVu.

翻译：自监督学习（SSL）算法通过建立自然图像不同部分之间的关联，能够生成有用的图像表征。然而在极端情况下，SSL模型可能非预期地记忆单个训练样本中的特定区域，而非学习有语义意义的关联。本研究系统性地探究了SSL模型中图像特异性信息的非预期记忆现象——我们称之为“既视感记忆”。具体而言，我们证明：给定训练好的模型以及仅包含背景（如水、天空、草地）的训练图像切片，可以高精度推断甚至视觉重建前景目标。此外，我们发现既视感记忆普遍存在于不同SSL算法中，且受特定设计选择加剧，同时无法通过传统表征质量评估技术检测。本研究对既视感记忆的探索揭示了SSL模型中此前未知的隐私风险，并提出了潜在的实用缓解策略。代码已开源至 https://github.com/facebookresearch/DejaVu。