There has been significant recent progress in training differentially private (DP) models which achieve accuracy that approaches the best non-private models. These DP models are typically pretrained on large public datasets and then fine-tuned on private downstream datasets that are relatively large and similar in distribution to the pretraining data. However, in many applications including personalization and federated learning, it is crucial to perform well (i) in the few-shot setting, as obtaining large amounts of labeled data may be problematic; and (ii) on datasets from a wide variety of domains for use in various specialist settings. To understand under which conditions few-shot DP can be effective, we perform an exhaustive set of experiments that reveals how the accuracy and vulnerability to attack of few-shot DP image classification models are affected as the number of shots per class, privacy level, model architecture, downstream dataset, and subset of learnable parameters in the model vary. We show that to achieve DP accuracy on par with non-private models, the shots per class must be increased as the privacy level increases by as much as 20 - 35$\times$ at $\epsilon=1$. We also show that learning parameter-efficient FiLM adapters under DP is competitive with and often superior to learning just the final classifier layer or learning all of the network parameters. Finally, we evaluate DP federated learning systems and establish state-of-the-art performance on the challenging FLAIR benchmark.

翻译：近期，差分隐私（DP）模型在训练方面取得了显著进展，其准确率已接近最优的非私有模型。这些DP模型通常先在大型公开数据集上进行预训练，然后在与预训练数据分布相似且规模较大的私有下游数据集上进行微调。然而，在个性化学习和联邦学习等众多应用中，关键挑战在于：（i）当获取大量标注数据存在困难时，需在小样本场景下表现良好；（ii）需适用于涵盖多种领域的专业场景数据集。为探究小样本DP有效的条件，我们通过一系列全面实验揭示了每类样本数、隐私保护强度、模型架构、下游数据集及模型可学习参数子集如何影响小样本DP图像分类模型的准确率与攻击脆弱性。研究表明，要使DP准确率与无隐私保护模型持平，当隐私预算ϵ=1时，每类样本数需根据隐私保护强度增加20-35倍。此外，在DP约束下学习参数高效的FiLM适配器，其性能可与仅学习最终分类层或全参数网络相媲美且往往更优。最终，我们评估了DP联邦学习系统，在具有挑战性的FLAIR基准测试上取得了当前最优性能。