The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection. While differential privacy (DP) is a prominent method to gauge the degree of security provided to the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when applying DP during the pre-training stage. Consequently, DP is yet not capable of protecting a substantial portion of the data used during the initial pre-training process. In this work, we first provide a theoretical understanding of the efficacy of DP training by analyzing the per-iteration loss improvement. We make a key observation that DP optimizers' performance degradation can be significantly mitigated by the use of limited public data, which leads to a novel DP continual pre-training strategy. Empirically, using only 10\% of public data, our strategy can achieve DP accuracy of 41.5\% on ImageNet-21k (with $\epsilon=8$), as well as non-DP accuracy of 55.7\% and and 60.0\% on downstream tasks Places365 and iNaturalist-2021, respectively, on par with state-of-the-art standard pre-training and substantially outperforming existing DP pre-trained models.

翻译：大型基础模型的卓越性能依赖于海量高质量数据的使用，而这些数据往往包含需要正式保护的敏感、私密及受版权保护内容。尽管差分隐私（DP）是衡量模型安全程度的显著方法，但由于在预训练阶段应用DP会导致性能下降，其应用通常局限于模型微调阶段。因此，DP尚无法保护初始预训练过程中使用的大部分数据。本文首先通过分析每轮迭代的损失改善，对DP训练的有效性提供理论理解。我们发现一个关键现象：使用有限公开数据可显著缓解DP优化器的性能退化，据此提出一种新颖的DP持续预训练策略。实验表明，该策略仅使用10%的公开数据，即可在ImageNet-21k数据集上（ε=8）达到41.5%的DP准确率，并在下游任务Places365和iNaturalist-2021上分别获得55.7%和60.0%的非DP准确率，其性能与当前最先进的标准预训练模型持平，同时显著优于现有DP预训练模型。