We investigate practical and scalable algorithms for training large language models (LLMs) with user-level differential privacy (DP) in order to provably safeguard all the examples contributed by each user. We study two variants of DP-SGD with: (1) example-level sampling (ELS) and per-example gradient clipping, and (2) user-level sampling (ULS) and per-user gradient clipping. We derive a novel user-level DP accountant that allows us to compute provably tight privacy guarantees for ELS. Using this, we show that while ELS can outperform ULS in specific settings, ULS generally yields better results when each user has a diverse collection of examples. We validate our findings through experiments in synthetic mean estimation and LLM fine-tuning tasks under fixed compute budgets. We find that ULS is significantly better in settings where either (1) strong privacy guarantees are required, or (2) the compute budget is large. Notably, our focus on LLM-compatible training algorithms allows us to scale to models with hundreds of millions of parameters and datasets with hundreds of thousands of users.

翻译：本文研究了在用户级差分隐私（DP）约束下训练大语言模型（LLM）的实用且可扩展算法，旨在可证明地保护每位用户贡献的所有示例。我们研究了两种DP-SGD变体：（1）采用示例级采样（ELS）和逐示例梯度裁剪；（2）采用用户级采样（ULS）和逐用户梯度裁剪。我们推导出一种新颖的用户级DP会计方法，使我们能够为ELS计算可证明的严格隐私保证。通过该方法，我们发现尽管ELS在特定场景下可能优于ULS，但当每位用户拥有多样化的示例集合时，ULS通常能产生更好的结果。我们在固定计算预算下，通过合成均值估计和LLM微调任务的实验验证了这些发现。我们发现，在以下任一情况下，ULS表现显著更优：（1）需要强隐私保证时，或（2）计算预算较大时。值得注意的是，我们对LLM兼容训练算法的关注，使得我们能够将模型规模扩展至数亿参数，并处理包含数十万用户的数据集。