We study few-shot Natural Language Understanding (NLU) tasks with Large Language Models (LLMs) in federated learning (FL) scenarios. It is a challenging task due to limited labeled data and communication capacities in FL, especially with mobile devices. Recent studies show LLMs can be prompted to perform few-shot NLU tasks like sentiment analysis and arithmetic reasoning. However, the huge sizes of LLMs result in high computation and communication costs, making classical FL schemes impractical. To address these challenges, we propose Low-Parameter Federated Learning (LP-FL). LP-FL combines few-shot prompt learning from LLMs with efficient communication and federating techniques. Our approach enables federated clients to assign soft labels to unlabeled data using gradually learned knowledge from the global model. Through iterative soft-label assigning, we continually expand the labeled set during the FL process. Additionally, to reduce computation and communication costs, LP-FL utilizes the Low-Rank Adaptation (LoRA) technique for compact learnable parameter construction, efficient local model fine-tuning, and affordable global model federation. LP-FL consistently outperforms Full-Parameter Federated Learning (FP-FL) in sentiment analysis tasks across various FL settings. Its resistance to overfitting allows LP-FL to equal or surpass centralized training in few-shot scenarios.

翻译：我们研究在联邦学习场景中利用大语言模型处理少样本自然语言理解任务。由于联邦学习中标记数据有限且通信能力受限（尤其在移动设备场景下），该任务具有挑战性。近期研究表明，大语言模型可通过提示学习执行情感分析和算术推理等少样本自然语言理解任务。然而，大语言模型的庞大规模导致计算与通信成本过高，使得传统联邦学习方案难以实际应用。为应对这些挑战，我们提出低参数量联邦学习。该方法将大语言模型的少样本提示学习与高效通信及联邦技术相结合，使联邦客户端能利用全局模型逐步习得的知识为未标记数据分配软标签。通过迭代式软标签分配流程，我们在联邦学习过程中持续扩充标记数据集。此外，为降低计算与通信成本，低参数量联邦学习采用低秩适配技术构建紧凑的可训练参数、实现高效的本地模型微调及可负担的全局模型联邦。在各种联邦学习设置下，低参数量联邦学习在情感分析任务中持续优于全参数联邦学习。其抗过拟合能力使低参数量联邦学习在少样本场景中能达到甚至超越集中式训练的性能表现。