Unsupervised pre-training is a common step in developing computer vision models and large language models. In this setting, the absence of labels requires the use of similarity-based loss functions, such as contrastive loss, that favor minimizing the distance between similar inputs and maximizing the distance between distinct inputs. As privacy concerns mount, training these models using differential privacy has become more important. However, due to how inputs are generated for these losses, one of their undesirable properties is that their $L_2$ sensitivity can grow with increasing batch size. This property is particularly disadvantageous for differentially private training methods, such as DP-SGD. To overcome this issue, we develop a new DP-SGD variant for similarity based loss functions -- in particular the commonly used contrastive loss -- that manipulates gradients of the objective function in a novel way to obtain a senstivity of the summed gradient that is $O(1)$ for batch size $n$. We test our DP-SGD variant on some preliminary CIFAR-10 pre-training and CIFAR-100 finetuning tasks and show that, in both tasks, our method's performance comes close to that of a non-private model and generally outperforms DP-SGD applied directly to the contrastive loss.

翻译：无监督预训练是开发计算机视觉模型和大语言模型的常见步骤。在此背景下，由于缺乏标签，需要使用基于相似度的损失函数（如对比损失），其倾向于最小化相似输入之间的距离，同时最大化不同输入之间的距离。随着隐私担忧日益加剧，采用差分隐私训练这些模型变得更为重要。然而，由于这些损失函数生成输入的方式，其一个不利特性是$L_2$敏感性会随批量大小增加而增长。这一特性对差分隐私训练方法（如DP-SGD）尤其不利。为解决此问题，我们针对基于相似度的损失函数（特别是常用的对比损失）开发了一种新的DP-SGD变体，该变体以新颖方式操纵目标函数的梯度，使得求和梯度的敏感性对于批量大小$n$为$O(1)$。我们在CIFAR-10预训练和CIFAR-100微调的初步任务上测试了该DP-SGD变体，结果表明，在两个任务中，我们的方法性能接近非私有模型，且普遍优于直接将DP-SGD应用于对比损失的方法。