Federated learning (FL) is increasingly deployed among multiple clients to train a shared model over decentralized data. To address privacy concerns, FL systems need to safeguard the clients' data from disclosure during training and control data leakage through trained models when exposed to untrusted domains. Distributed differential privacy (DP) offers an appealing solution in this regard as it achieves a balanced tradeoff between privacy and utility without a trusted server. However, existing distributed DP mechanisms are impractical in the presence of client dropout, resulting in poor privacy guarantees or degraded training accuracy. In addition, these mechanisms suffer from severe efficiency issues. We present Hyades, a distributed differentially private FL framework that is highly efficient and resilient to client dropout. Specifically, we develop a novel 'add-then-remove' scheme that enforces a required noise level precisely in each training round, even if some sampled clients drop out. This ensures that the privacy budget is utilized prudently, despite unpredictable client dynamics. To boost performance, Hyades operates as a distributed parallel architecture via encapsulating the communication and computation operations into stages. It automatically divides the global model aggregation into several chunk-aggregation tasks and pipelines them for optimal speedup. Large-scale deployment evaluations demonstrate that Hyades efficiently handles client dropout in various realistic FL scenarios, achieving the optimal privacy-utility tradeoff and accelerating training by up to 2.4$\times$ compared to existing solutions.

翻译：联邦学习（FL）正日益部署在多个客户端之间，用于在分散数据上训练共享模型。为解决隐私问题，联邦学习系统需要保护客户端数据在训练过程中不被泄露，并控制通过训练模型在暴露于不可信域时的数据泄漏。分布式差分隐私（DP）提供了一种有吸引力的解决方案，因为它能在无信任服务器的情况下实现隐私与效用之间的平衡权衡。然而，现有的分布式差分隐私机制在存在客户端丢失时缺乏实用性，导致隐私保证差或训练精度下降。此外，这些机制还存在严重的效率问题。我们提出Hyades，一个高效且对客户端丢失具有鲁棒性的分布式差分隐私联邦学习框架。具体地，我们开发了一种新颖的“先加后减”方案，即使部分采样客户端丢失，也能在每个训练轮次中精确施加所需噪声水平。这确保了隐私预算得到审慎利用，尽管客户端动态不可预测。为提升性能，Hyades通过将通信与计算操作封装成阶段，采用分布式并行架构运行。它自动将全局模型聚合划分为多个块聚合任务，并将其流水线化以实现最优加速。大规模部署评估表明，Hyades能在各种真实联邦学习场景中高效处理客户端丢失，实现最优的隐私-效用权衡，并相比现有解决方案加速训练高达2.4倍。