Artificial intelligence (AI) has seen a tremendous surge in capabilities thanks to the use of foundation models trained on internet-scale data. On the flip side, the uncurated nature of internet-scale data also poses significant privacy and legal risks, as they often contain personal information or copyrighted material that should not be trained on without permission. In this work, we propose as a mitigation measure a recipe to train foundation vision models with differential privacy (DP) guarantee. We identify masked autoencoders as a suitable learning algorithm that aligns well with DP-SGD, and train ViP -- a Vision transformer with differential Privacy -- under a strict privacy budget of $\epsilon=8$ on the LAION400M dataset. We evaluate the quality of representation learned by ViP using standard downstream vision tasks; in particular, ViP achieves a (non-private) linear probing accuracy of $55.7\%$ on ImageNet, comparable to that of end-to-end trained AlexNet (trained and evaluated on ImageNet). Our result suggests that scaling to internet-scale data can be practical for private learning. Code is available at \url{https://github.com/facebookresearch/ViP-MAE}.

翻译：人工智能（AI）得益于使用互联网规模数据训练的基础模型，其能力得到了显著提升。然而，互联网规模数据的无筛选特性也带来了严重的隐私与法律风险——这些数据常包含未经许可不得用于训练的个人信息或受版权保护的内容。为缓解这一问题，本文提出了一种具有差分隐私（DP）保证的基础视觉模型训练方案。我们识别出掩码自编码器是一种与DP-SGD高度兼容的学习算法，并在LAION400M数据集上以严格的隐私预算$\epsilon=8$训练了ViP——一种具有差分隐私保护的视觉Transformer。通过标准下游视觉任务评估ViP学习到的表征质量：在ImageNet上，ViP实现了$55.7\%$的（非隐私）线性探测准确率，与端到端训练（在ImageNet上训练并评估）的AlexNet性能相当。这一结果表明，将私有学习扩展至互联网规模数据具有可行性。代码已开源至\url{https://github.com/facebookresearch/ViP-MAE}。