Federated learning (FL) allows clients in an Internet of Things (IoT) system to collaboratively train a global model without sharing their local data with a server. However, clients' contributions to the server can still leak sensitive information. Differential privacy (DP) addresses such leakage by providing formal privacy guarantees, with mechanisms that add randomness to the clients' contributions. The randomness makes it infeasible to train large transformer-based models, common in modern IoT systems. In this work, we empirically evaluate the practicality of fine-tuning large scale on-device transformer-based models with differential privacy in a federated learning system. We conduct comprehensive experiments on various system properties for tasks spanning a multitude of domains: speech recognition, computer vision (CV) and natural language understanding (NLU). Our results show that full fine-tuning under differentially private federated learning (DP-FL) generally leads to huge performance degradation which can be alleviated by reducing the dimensionality of contributions through parameter-efficient fine-tuning (PEFT). Our benchmarks of existing DP-PEFT methods show that DP-Low-Rank Adaptation (DP-LoRA) consistently outperforms other methods. An even more promising approach, DyLoRA, which makes the low rank variable, when naively combined with FL would straightforwardly break differential privacy. We therefore propose an adaptation method that can be combined with differential privacy and call it DP-DyLoRA. Finally, we are able to reduce the accuracy degradation and word error rate (WER) increase due to DP to less than 2% and 7% respectively with 1 million clients and a stringent privacy budget of {\epsilon}=2.

翻译：联邦学习（FL）使得物联网（IoT）系统中的客户端能够协作训练全局模型，而无需向服务器共享本地数据。然而，客户端对服务器的贡献仍可能泄露敏感信息。差分隐私（DP）通过为客户端贡献添加随机性的机制，提供形式化的隐私保证，从而应对此类泄露。随机性使得训练现代物联网系统中常见的大规模基于Transformer的模型变得不可行。在本工作中，我们实证评估了在联邦学习系统中使用差分隐私在设备端微调大规模Transformer模型的实用性。我们在涵盖语音识别、计算机视觉（CV）和自然语言理解（NLU）等多个领域的任务上，针对各种系统特性进行了全面实验。结果表明，在差分隐私联邦学习（DP-FL）下进行全参数微调通常会导致严重的性能下降，而通过参数高效微调（PEFT）降低贡献维度可缓解此问题。我们对现有DP-PEFT方法的基准测试表明，DP-低秩适配（DP-LoRA）始终优于其他方法。一种更具前景的方法DyLoRA（使低秩可变）若直接与FL结合，将明确破坏差分隐私。因此，我们提出一种可与差分隐私结合的适配方法，称为DP-DyLoRA。最终，在100万客户端且严格隐私预算{\epsilon}=2的条件下，我们将因DP导致的准确率下降和词错误率（WER）上升分别控制在2%和7%以内。