Artificial intelligence (AI) has seen a tremendous surge in capabilities thanks to the use of foundation models trained on internet-scale data. On the flip side, the uncurated nature of internet-scale data also poses significant privacy and legal risks, as they often contain personal information or copyrighted material that should not be trained on without permission. In this work, we propose as a mitigation measure a recipe to train foundation vision models with differential privacy (DP) guarantee. We identify masked autoencoders as a suitable learning algorithm that aligns well with DP-SGD, and train ViP -- a Vision transformer with differential Privacy -- under a strict privacy budget of $\epsilon=8$ on the LAION400M dataset. We evaluate the quality of representation learned by ViP using standard downstream vision tasks; in particular, ViP achieves a (non-private) linear probing accuracy of $55.7\%$ on ImageNet, comparable to that of end-to-end trained AlexNet (trained and evaluated on ImageNet). Our result suggests that scaling to internet-scale data can be practical for private learning.

翻译：摘要：人工智能(AI)的能力因使用基于互联网规模数据训练的基础模型而获得巨大提升。然而，互联网规模数据的非策划特性也带来显著的隐私和法律风险，因为这些数据通常包含未经授权不应被用于训练的隐私信息或受版权保护的内容。本文提出一种缓解措施——一种在差分隐私(DP)保证下训练基础视觉模型的方案。我们确定掩码自编码器是一种与DP-SGD高度兼容的合适学习算法，并在LAION400M数据集上以严格隐私预算$\epsilon=8$训练了ViP（具有差分隐私的视觉Transformer）。通过标准下游视觉任务评估ViP学习到的表征质量：具体而言，ViP在ImageNet上达到（非隐私）线性探测准确率55.7%，与端到端训练的AlexNet（在ImageNet上训练并评估）性能相当。我们的结果表明，将模型扩展至互联网规模数据对隐私学习具有实践可行性。