Correlated noise mechanisms such as DP Matrix Factorization (DP-MF) have proven to be effective alternatives to DP-SGD in large-epsilon few-epoch training regimes. Significant work has been done to find the best correlated noise strategies, and the current state-of-the-art approach is DP-BandMF, which optimally balances the benefits of privacy amplification and noise correlation. Despite it's utility advantages, severe scalability limitations prevent this mechanism from handling large-scale training scenarios where the number of training iterations may exceed $10^4$ and the number of model parameters may exceed $10^7$. In this work, we present techniques to scale up DP-BandMF along these two dimensions, significantly extending it's reach and enabling it to handle settings with virtually any number of model parameters and training iterations, with negligible utility degradation.

翻译：在大ε值、少轮次训练场景中，相关噪声机制（如差分隐私矩阵分解机制）已被证明是替代差分隐私随机梯度下降的有效方案。现有研究已对最优相关噪声策略进行了大量探索，当前最先进的方法为差分隐私带状矩阵分解机制，该机制在隐私放大与噪声关联的效益间实现了最优平衡。尽管该机制具有显著效用优势，但其严重的可扩展性限制使其无法适应训练迭代次数可能超过$10^4$、模型参数量可能超过$10^7$的大规模训练场景。本研究提出了沿这两个维度扩展差分隐私带状矩阵分解机制的技术方案，显著拓展了其适用范围，使其能够处理任意规模的模型参数和训练迭代场景，同时保持可忽略的效用损失。