Federated learning (FL) enables training of a global model while keeping raw data on end-devices. Despite this, FL has shown to leak private user information and thus in practice, it is often coupled with methods such as differential privacy (DP) and secure vector sum to provide formal privacy guarantees to its participants. In realistic cross-device deployments, the data are highly heterogeneous, so vanilla federated learning converges slowly and generalizes poorly. Clustered federated learning (CFL) mitigates this by segregating users into clusters, leading to lower intra-cluster data heterogeneity. Nevertheless, coupling CFL with DP remains challenging: the injected DP noise makes individual client updates excessively noisy, and the server is unable to initialize cluster centroids with the less noisy aggregated updates. To address this challenge, we propose PINA, a two-stage framework that first lets each client fine-tune a lightweight low-rank adaptation (LoRA) adapter and privately share a compressed sketch of the update. The server leverages these sketches to construct robust cluster centroids. In the second stage, PINA introduces a normality-driven aggregation mechanism that improves convergence and robustness. Our method retains the benefits of clustered FL while providing formal privacy guarantees against an untrusted server. Extensive evaluations show that our proposed method outperforms state-of-the-art DP-FL algorithms by an average of 2.9% in accuracy for privacy budgets (epsilon in {2, 8}).

翻译：联邦学习（FL）支持在将原始数据保留在终端设备上的同时训练全局模型。尽管如此，联邦学习已被证明会泄露用户隐私信息，因此在实践中常与差分隐私（DP）和安全向量求和等方法结合使用，为参与者提供形式化隐私保障。在真实的跨设备部署场景中，数据高度异构，导致标准联邦学习收敛速度慢且泛化能力差。聚类联邦学习（CFL）通过将用户划分为不同聚类来缓解这一问题，从而降低聚类内部的数据异构性。然而，将CFL与DP结合仍面临挑战：注入的DP噪声会使单个客户端更新过度噪声化，且服务器无法利用噪声较小的聚合更新来初始化聚类中心。为应对这一挑战，我们提出PINA框架，该两阶段框架首先让每个客户端微调轻量级低秩自适应（LoRA）适配器，并私密共享更新的压缩摘要。服务器利用这些摘要构建鲁棒的聚类中心。在第二阶段，PINA引入正态性驱动聚合机制，提升收敛速度与鲁棒性。我们的方法既保留了聚类联邦学习的优势，又为不可信服务器提供了形式化隐私保障。大量评估表明，在隐私预算（ε∈{2,8}）条件下，本方法在准确率上平均比当前最优的DP-FL算法高出2.9%。