Matrix factorization (MF) mechanisms for differential privacy (DP) have substantially improved the state-of-the-art in privacy-utility-computation tradeoffs for ML applications in a variety of scenarios, but in both the centralized and federated settings there remain instances where either MF cannot be easily applied, or other algorithms provide better tradeoffs (typically, as $\epsilon$ becomes small). In this work, we show how MF can subsume prior state-of-the-art algorithms in both federated and centralized training settings, across all privacy budgets. The key technique throughout is the construction of MF mechanisms with banded matrices (lower-triangular matrices with at most $\hat{b}$ nonzero bands including the main diagonal). For cross-device federated learning (FL), this enables multiple-participations with a relaxed device participation schema compatible with practical FL infrastructure (as demonstrated by a production deployment). In the centralized setting, we prove that banded matrices enjoy the same privacy amplification results as the ubiquitous DP-SGD algorithm, but can provide strictly better performance in most scenarios -- this lets us always at least match DP-SGD, and often outperform it.

翻译：针对差分隐私（DP）的矩阵分解（MF）机制已在多种场景下显著提升了机器学习应用中隐私-效用-计算权衡的性能前沿，但在集中式和联邦设置中仍存在MF难以直接应用，或其他算法（通常当$\epsilon$变小时）能提供更优权衡的情况。本研究展示了MF如何在所有隐私预算下，同时在联邦和集中训练场景中涵盖先前最先进的算法。贯穿始终的关键技术是构建具有带状矩阵（最多包含$\hat{b}$条非零带（含主对角线）的下三角矩阵）的MF机制。对于跨设备联邦学习（FL），这种方法能够以兼容实际FL基础设施（经生产部署验证）的宽松设备参与模式实现多重参与。在集中式设置中，我们证明带状矩阵享有与普遍使用的DP-SGD算法相同的隐私放大效果，但在多数场景下能提供严格更优的性能——这使我们始终至少能达到DP-SGD的水平，且常常优于它。