There has been significant recent progress in training differentially private (DP) models which achieve accuracy that approaches the best non-private models. These DP models are typically pretrained on large public datasets and then fine-tuned on downstream datasets that are (i) relatively large, and (ii) similar in distribution to the pretraining data. However, in many applications including personalization, it is crucial to perform well in the few-shot setting, as obtaining large amounts of labeled data may be problematic; and on images from a wide variety of domains for use in various specialist settings. To understand under which conditions few-shot DP can be effective, we perform an exhaustive set of experiments that reveals how the accuracy and vulnerability to attack of few-shot DP image classification models are affected as the number of shots per class, privacy level, model architecture, dataset, and subset of learnable parameters in the model vary. We show that to achieve DP accuracy on par with non-private models, the shots per class must be increased as the privacy level increases by as much as 32$\times$ for CIFAR-100 at $\epsilon=1$. We also find that few-shot non-private models are highly susceptible to membership inference attacks. DP provides clear mitigation against the attacks, but a small $\epsilon$ is required to effectively prevent them. Finally, we evaluate DP federated learning systems and establish state-of-the-art performance on the challenging FLAIR federated learning benchmark.

翻译：近年来，在训练差分隐私（DP）模型方面取得了显著进展，这些模型的精度已接近最优的非隐私模型。这些DP模型通常在大规模公开数据集上进行预训练，然后在（i）规模相对较大且（ii）与预训练数据分布相似的的下游数据集上进行微调。然而，在包括个性化在内的许多应用中，在小样本场景下取得良好性能至关重要，因为获取大量标注数据可能存在问题；同时，针对各种专业场景中来自广泛领域的图像，保持性能同样关键。为了理解小样本DP在何种条件下有效，我们进行了一组详尽的实验，揭示了每类样本数、隐私水平、模型架构、数据集以及模型中可学习参数子集的变化如何影响小样本DP图像分类模型的准确性和对攻击的脆弱性。我们发现，为了实现与非隐私模型相当的DP精度，每类样本数必须随隐私水平的提高而增加，对于CIFAR-100数据集在ϵ=1时，所需样本数需增加多达32倍。我们还发现，非隐私的小样本模型极易受到成员推断攻击。DP对此类攻击提供了明确的缓解措施，但需要较小的ϵ才能有效阻止攻击。最后，我们评估了DP联邦学习系统，并在具有挑战性的FLAIR联邦学习基准上建立了最先进的性能。