In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.

翻译：本文探讨了大型模型的联邦定制化，并强调了其在联邦学习框架下所面临的关键挑战。我们回顾了若干主流的大型模型定制化技术，包括全参数微调、高效微调、提示工程、前缀调优、知识蒸馏以及检索增强生成。随后，我们讨论了这些技术如何在联邦学习框架内实施。此外，我们进行了联邦前缀调优的实验，据我们所知，这是首次在联邦学习环境中应用前缀调优的尝试。实验验证了其可行性，其性能接近集中式方法。与其他三种联邦定制化方法的进一步比较表明，该方法具有竞争力的性能、令人满意的效率以及一致的鲁棒性。