Decentralized optimization is gaining increased traction due to its widespread applications in large-scale machine learning and multi-agent systems. The same mechanism that enables its success, i.e., information sharing among participating agents, however, also leads to the disclosure of individual agents' private information, which is unacceptable when sensitive data are involved. As differential privacy is becoming a de facto standard for privacy preservation, recently results have emerged integrating differential privacy with distributed optimization. However, directly incorporating differential privacy design in existing distributed optimization approaches significantly compromises optimization accuracy. In this paper, we propose to redesign and tailor gradient methods for differentially-private distributed optimization, and propose two differential-privacy oriented gradient methods that can ensure both rigorous epsilon-differential privacy and optimality. The first algorithm is based on static-consensus based gradient methods, and the second algorithm is based on dynamic-consensus (gradient-tracking) based distributed optimization methods and, hence, is applicable to general directed interaction graph topologies. Both algorithms can simultaneously ensure almost sure convergence to an optimal solution and a finite privacy budget, even when the number of iterations goes to infinity. To our knowledge, this is the first time that both goals are achieved simultaneously. Numerical simulations using a distributed estimation problem and experimental results on a benchmark dataset confirm the effectiveness of the proposed approaches.

翻译：去中心化优化因其在大型机器学习与多智能体系统中的广泛应用而日益受到关注。然而，其成功所依赖的机制——即参与智能体之间的信息共享——也导致了个体智能体隐私信息的泄露，这在涉及敏感数据时是不可接受的。随着差分隐私逐渐成为隐私保护的默认标准，近期已有研究开始将差分隐私与分布式优化相结合。但直接将差分隐私设计纳入现有分布式优化方法会显著降低优化精度。本文旨在为差分隐私分布式优化重新设计和定制梯度方法，提出两种面向差分隐私的梯度方法，既能保证严格的ε-差分隐私性，又能确保最优性。第一种算法基于静态共识梯度方法，第二种算法基于动态共识（梯度追踪）分布式优化方法，因此适用于一般有向交互图拓扑。两种算法即使在迭代次数趋于无穷的情况下，也能同时保证几乎必然收敛到最优解并维持有限隐私预算。据我们所知，这是首次实现这两种目标的同步达成。基于分布式估计问题的数值仿真以及基准数据集上的实验结果验证了所提方法的有效性。