We address the challenge of ensuring differential privacy (DP) guarantees in training deep retrieval systems. Training these systems often involves the use of contrastive-style losses, which are typically non-per-example decomposable, making them difficult to directly DP-train with since common techniques require per-example gradients. To address this issue, we propose an approach that prioritizes ensuring query privacy prior to training a deep retrieval system. Our method employs DP language models (LMs) to generate private synthetic queries representative of the original data. These synthetic queries can be used in downstream retrieval system training without compromising privacy. Our approach demonstrates a significant enhancement in retrieval quality compared to direct DP-training, all while maintaining query-level privacy guarantees. This work highlights the potential of harnessing LMs to overcome limitations in standard DP-training methods.

翻译：我们致力于解决在训练深度检索系统时确保差分隐私（DP）保证的挑战。训练这些系统通常涉及使用对比式损失函数，这类损失通常不具备按样本可分解性，使得直接进行DP训练变得困难，因为常见技术需要按样本梯度。为解决此问题，我们提出一种在训练深度检索系统之前优先确保查询隐私的方法。我们的方法采用DP语言模型（LMs）来生成代表原始数据的私有合成查询。这些合成查询可用于下游检索系统训练，且不会损害隐私。与直接DP训练相比，我们的方法在保持查询级隐私保证的同时，显著提升了检索质量。这项工作凸显了利用LMs克服标准DP训练方法局限性的潜力。