Fine-tuning (FT) large language models (LLMs) is crucial for adapting general-purpose models to specific tasks, enhancing accuracy and relevance with minimal resources. To further enhance generalization ability while reducing training costs, this paper proposes Federated LoRA with Dropout (FedLoDrop), a new framework that applies dropout to the rows and columns of the trainable matrix in Federated LoRA. A generalization error bound and convergence analysis under sparsity regularization are obtained, which elucidate the fundamental trade-off between underfitting and overfitting. The error bound reveals that a higher dropout rate increases model sparsity, thereby lowering the upper bound of pointwise hypothesis stability (PHS). While this reduces the gap between empirical and generalization errors, it also incurs a higher empirical error, which, together with the gap, determines the overall generalization error. On the other hand, though dropout reduces communication costs, deploying FedLoDrop at the network edge still faces challenges due to limited network resources. To address this issue, an optimization problem is formulated to minimize the upper bound of the generalization error, by jointly optimizing the dropout rate and resource allocation subject to the latency and per-device energy consumption constraints. To solve this problem, a branch-and-bound (B\&B)-based method is proposed to obtain its globally optimal solution. Moreover, to reduce the high computational complexity of the B\&B-based method, a penalized successive convex approximation (P-SCA)-based algorithm is proposed to efficiently obtain its high-quality suboptimal solution. Finally, numerical results demonstrate the effectiveness of the proposed approach in mitigating overfitting and improving the generalization capability.

翻译：微调（FT）大型语言模型（LLM）对于将通用模型适配至特定任务至关重要，能够以最小资源提升准确性与相关性。为进一步增强泛化能力并降低训练成本，本文提出基于Dropout的联邦LoRA（FedLoDrop），该新框架在联邦LoRA的可训练矩阵行列上应用Dropout技术。通过稀疏正则化条件下的泛化误差界与收敛性分析，揭示了欠拟合与过拟合之间的本质权衡。误差界表明：较高的Dropout率会增强模型稀疏性，从而降低逐点假设稳定性（PHS）的上界；这虽能缩小经验误差与泛化误差间的差距，但也会导致更高的经验误差——二者共同决定了整体泛化误差。另一方面，尽管Dropout降低了通信开销，在网络边缘部署FedLoDrop仍因有限网络资源面临挑战。为此，本文构建了在时延与单设备能耗约束下联合优化Dropout率与资源分配的优化问题，以最小化泛化误差上界。针对该问题，提出基于分支定界（B&B）的方法获取全局最优解。为降低B&B方法的高计算复杂度，进一步提出基于惩罚连续凸近似（P-SCA）的算法以高效获得高质量次优解。最终，数值结果验证了所提方法在抑制过拟合与提升泛化能力方面的有效性。