Small on-device models have been successfully trained with user-level differential privacy (DP) for next word prediction and image classification tasks in the past. However, existing methods can fail when directly applied to learn embedding models using supervised training data with a large class space. To achieve user-level DP for large image-to-embedding feature extractors, we propose DP-FedEmb, a variant of federated learning algorithms with per-user sensitivity control and noise addition, to train from user-partitioned data centralized in the datacenter. DP-FedEmb combines virtual clients, partial aggregation, private local fine-tuning, and public pretraining to achieve strong privacy utility trade-offs. We apply DP-FedEmb to train image embedding models for faces, landmarks and natural species, and demonstrate its superior utility under same privacy budget on benchmark datasets DigiFace, EMNIST, GLD and iNaturalist. We further illustrate it is possible to achieve strong user-level DP guarantees of $\epsilon<4$ while controlling the utility drop within 5%, when millions of users can participate in training.

翻译：过去，在下一词预测和图像分类任务中，已成功使用用户级差分隐私（DP）训练小型设备端模型。然而，现有方法在直接应用于通过具有大类空间的监督训练数据学习嵌入模型时可能失效。为了实现大型图像到嵌入特征提取器的用户级差分隐私，我们提出DP-FedEmb——一种基于联邦学习算法的变体，通过对每个用户进行灵敏度控制并添加噪声，从数据中心集中管理的用户分区数据中进行训练。DP-FedEmb结合了虚拟客户端、部分聚合、私有本地微调和公共预训练，以实现强隐私效用权衡。我们将DP-FedEmb应用于人脸、地标及自然物种图像嵌入模型的训练，并在基准数据集DigiFace、EMNIST、GLD和iNaturalist上展示了其在相同隐私预算下更优的效用。我们进一步证明，当数百万用户参与训练时，有可能在将效用下降控制在5%以内的同时，实现强用户级DP保障（$\epsilon<4$）。