We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data. Existing work has demonstrated that high accuracy is possible under strong privacy constraint, yet requires significant computational overhead or modifications to the network architecture. We propose differentially private bias-term fine-tuning (DP-BiTFiT), which matches the state-of-the-art accuracy for DP algorithms and the efficiency of the standard BiTFiT. DP-BiTFiT is model agnostic (not modifying the network architecture), parameter efficient (only training about 0.1% of the parameters), and computation efficient (almost removing the overhead caused by DP, in both the time and space complexity). On a wide range of tasks, DP-BiTFiT is 2~30X faster and uses 2~8X less memory than DP full fine-tuning, even faster than the standard full fine-tuning. This amazing efficiency enables us to conduct DP fine-tuning on language and vision tasks with long-sequence texts and high-resolution images, which were computationally difficult using existing methods. We open-source our code at FastDP (https://github.com/awslabs/fast-differential-privacy).

翻译：我们研究了大规模预训练模型的差分隐私（DP）微调问题——这是一种适用于处理敏感数据下游任务的近期隐私保护方法。现有研究表明，在强隐私约束下实现高精度是可能的，但需要显著的计算开销或对网络架构进行修改。我们提出了差分隐私偏置项微调（DP-BiTFiT），该方法在精度上达到了DP算法的先进水平，同时保持了标准BiTFiT的效率。DP-BiTFiT具有模型无关性（不修改网络架构）、参数高效性（仅训练约0.1%的参数）和计算高效性（在时间和空间复杂度上几乎消除了DP引入的开销）。在广泛的任务中，DP-BiTFiT比DP全参数微调快2~30倍且内存使用减少2~8倍，甚至比标准全参数微调更快。这种卓越的效率使我们能够在长文本序列和高分辨率图像的视觉与语言任务上实现DP微调，而使用现有方法进行此类任务在计算上曾十分困难。我们在FastDP（https://github.com/awslabs/fast-differential-privacy）开源了代码。