We study gradient descent under linearly correlated noise. Our work is motivated by recent practical methods for optimization with differential privacy (DP), such as DP-FTRL, which achieve strong performance in settings where privacy amplification techniques are infeasible (such as in federated learning). These methods inject privacy noise through a matrix factorization mechanism, making the noise linearly correlated over iterations. We propose a simplified setting that distills key facets of these methods and isolates the impact of linearly correlated noise. We analyze the behavior of gradient descent in this setting, for both convex and non-convex functions. Our analysis is demonstrably tighter than prior work and recovers multiple important special cases exactly (including anticorrelated perturbed gradient descent). We use our results to develop new, effective matrix factorizations for differentially private optimization, and highlight the benefits of these factorizations theoretically and empirically.

翻译：我们研究了在线性相关噪声下的梯度下降方法。本研究受近期在差分隐私优化中取得显著成效的实践方法（如DP-FTRL）启发，这些方法在隐私放大技术不可行（例如联邦学习场景）时表现优异。此类方法通过矩阵分解机制注入隐私噪声，导致噪声在迭代过程中呈现线性相关性。我们提出一个简化设定，提炼出这些方法的关键特征，并分离出线性相关噪声的独立影响。针对凸函数与非凸函数，我们分析了该设定下梯度下降的行为特性。我们的分析相较于先前工作具有可证明的紧致性，并精确恢复了多种重要特例（包括反相关扰动梯度下降）。基于研究结果，我们开发了面向差分隐私优化的新型高效矩阵分解方法，并从理论与实证两个维度揭示了这些分解方案的优越性。