Matrix factorization (MF) mechanisms for differential privacy (DP) have substantially improved the state-of-the-art in privacy-utility-computation tradeoffs for ML applications in a variety of scenarios, but in both the centralized and federated settings there remain instances where either MF cannot be easily applied, or other algorithms provide better tradeoffs (typically, as $\epsilon$ becomes small). In this work, we show how MF can subsume prior state-of-the-art algorithms in both federated and centralized training settings, across all privacy budgets. The key technique throughout is the construction of MF mechanisms with banded matrices. For cross-device federated learning (FL), this enables multiple-participations with a relaxed device participation schema compatible with practical FL infrastructure (as demonstrated by a production deployment). In the centralized setting, we prove that banded matrices enjoy the same privacy amplification results as for the ubiquitous DP-SGD algorithm, but can provide strictly better performance in most scenarios -- this lets us always at least match DP-SGD, and often outperform it even at $\epsilon\ll2$. Finally, $\hat{b}$-banded matrices substantially reduce the memory and time complexity of per-step noise generation from $\mathcal{O}(n)$, $n$ the total number of iterations, to a constant $\mathcal{O}(\hat{b})$, compared to general MF mechanisms.

翻译：用于差分隐私（DP）的矩阵分解（MF）机制在多种场景下显著提升了机器学习应用中隐私-效用-计算权衡的前沿水平，但在集中式和联邦设置中仍存在一些问题：要么MF无法轻易应用，要么其他算法能提供更好的权衡（通常在ε变小时）。本文展示了MF如何能够在所有隐私预算下，同时涵盖联邦和集中式训练设置中先前最先进的算法。贯穿全文的关键技术是构建具有带状矩阵的MF机制。针对跨设备联邦学习（FL），该方法支持一种与实用FL基础设施兼容的宽松设备参与模式下的多次参与（已通过生产部署验证）。在集中式设置中，我们证明了带状矩阵与普遍使用的DP-SGD算法享有相同的隐私放大结果，但在大多数场景下能提供严格更优的性能——这使得我们至少能始终匹配DP-SGD，并且在ε≪2时往往能超越它。最后，与通用MF机制相比，b-带状矩阵将每步噪声生成的内存和时间复杂度从O(n)（n为总迭代次数）大幅降低至常数O(b)。