Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.

翻译：联邦学习作为一种保护隐私的训练范式已得到广泛研究。近年来，联邦分块坐标下降法在训练大规模模型中成为热门选择，因为它允许客户端本地仅训练模型子集而非完整模型。然而，在大语言模型时代，即便是单个分块也可能包含大量参数，这导致了显著通信延迟，尤其对于资源受限的客户端。为了解决联邦训练/微调大语言模型中的这一挑战，我们提出ParaBlock，一种建立通信与计算两个并行线程以增强通信效率的新型方法。我们理论证明了ParaBlock能够达到与标准联邦分块坐标下降法相同的收敛速率。在通用指令遵循与数学推理任务上微调大语言模型的实证评估证实，ParaBlock不仅保持了强劲性能，还显著提升了通信效率。