Federated Learning (FL) offers a promising solution for training machine learning models across distributed data sources while preserving data privacy. However, FL faces critical challenges related to communication overhead and local resource constraints, especially in the era of Large Language Models (LLMs) with billions of parameters. The sheer size of these models exacerbates both memory and communication constraints, making efficient transmission and processing essential for practical deployment. NVIDIA FLARE, an open-source SDK for federated learning, addresses these challenges by introducing advanced communication capabilities. Building upon existing solutions for large object streaming, we enhance FL workflows for LLMs through two key techniques: message quantization and container/file streaming. Quantization reduces message size, while streaming enables efficient memory management, improving scalability and integration with existing workflows. These advancements significantly enhance the robustness and efficiency of FL with LLMs, ensuring better performance in real-world federated learning scenarios.

翻译：联邦学习（Federated Learning, FL）为在分布式数据源上训练机器学习模型提供了一种有前景的解决方案，同时能保护数据隐私。然而，联邦学习面临着通信开销和本地资源限制等关键挑战，尤其是在拥有数十亿参数的大语言模型（Large Language Models, LLMs）时代。这些模型的庞大规模加剧了内存和通信限制，使得高效的传输与处理对于实际部署至关重要。NVIDIA FLARE，一个用于联邦学习的开源软件开发工具包，通过引入先进的通信能力来应对这些挑战。在现有的大对象流式传输解决方案基础上，我们通过两项关键技术增强了面向LLMs的联邦学习工作流：消息量化与容器/文件流式传输。量化减少了消息大小，而流式传输实现了高效的内存管理，从而提升了可扩展性以及与现有工作流的集成度。这些进展显著增强了基于LLMs的联邦学习的鲁棒性和效率，确保了在真实联邦学习场景中更好的性能。