We study differentially private distributed optimization under communication constraints. A server using SGD for optimization aggregates the client-side local gradients for model updates using distributed mean estimation (DME). We develop a communication-efficient private DME, using the recently developed multi-message shuffled (MMS) privacy framework. We analyze our proposed DME scheme to show that it achieves the order-optimal privacy-communication-performance tradeoff resolving an open question in [1], whether the shuffled models can improve the tradeoff obtained in Secure Aggregation. This also resolves an open question on the optimal trade-off for private vector sum in the MMS model. We achieve it through a novel privacy mechanism that non-uniformly allocates privacy at different resolutions of the local gradient vectors. These results are directly applied to give guarantees on private distributed learning algorithms using this for private gradient aggregation iteratively. We also numerically evaluate the private DME algorithms.

翻译：我们研究了通信约束下的差分隐私分布式优化问题。服务器使用随机梯度下降进行优化时，通过分布式均值估计聚合客户端局部梯度以更新模型。我们利用最新提出的多消息混洗（MMS）隐私框架，开发了一种通信高效的私有化分布式均值估计方案。通过分析所提出的分布式均值估计方案，我们证明其实现了阶最优的隐私-通信-性能权衡，从而解决了[1]中提出的公开问题——混洗模型能否改善安全聚合中获得的权衡。这一成果也解决了多消息混洗模型中私有向量求和最优权衡的公开问题。我们通过一种新颖的隐私机制实现这一目标，该机制在局部梯度向量的不同分辨率上非均匀分配隐私预算。这些结果可直接应用于通过迭代私有梯度聚合来保证私有分布式学习算法的性能。最后，我们对私有分布式均值估计算法进行了数值评估。