While ``instruction-tuned" generative large language models (LLMs) have demonstrated an impressive ability to generalize to new tasks, the training phases heavily rely on large amounts of diverse and high-quality instruction data (such as ChatGPT and GPT-4). Unfortunately, acquiring high-quality data, especially when it comes to human-written data, can pose significant challenges both in terms of cost and accessibility. Moreover, concerns related to privacy can further limit access to such data, making the process of obtaining it a complex and nuanced undertaking. Consequently, this hinders the generality of the tuned models and may restrict their effectiveness in certain contexts. To tackle this issue, our study introduces a new approach called Federated Instruction Tuning (FedIT), which leverages federated learning (FL) as the learning framework for the instruction tuning of LLMs. This marks the first exploration of FL-based instruction tuning for LLMs. This is especially important since text data is predominantly generated by end users. Therefore, it is imperative to design and adapt FL approaches to effectively leverage these users' diverse instructions stored on local devices, while preserving privacy and ensuring data security. In the current paper, by conducting widely used GPT-4 auto-evaluation, we demonstrate that by exploiting the heterogeneous and diverse sets of instructions on the client's end with the proposed framework FedIT, we improved the performance of LLMs compared to centralized training with only limited local instructions. Further, in this paper, we developed a Github repository named Shepherd. This repository offers a foundational framework for exploring federated fine-tuning of LLMs using heterogeneous instructions across diverse categories.

翻译：尽管“指令微调”的生成式大型语言模型（LLMs）已展现出对新任务令人印象深刻的泛化能力，但其训练阶段高度依赖大量多样化且高质量的指令数据（如ChatGPT和GPT-4）。然而，获取高质量数据——尤其是人工撰写的数据——在成本和可及性方面均面临显著挑战。此外，与隐私相关的担忧进一步限制了此类数据的访问，使得获取数据成为一项复杂而微妙的工程。这最终阻碍了微调模型的通用性，并可能限制其在特定场景中的有效性。为解决这一问题，本研究提出了一种名为“联邦指令微调”（FedIT）的新方法，该方法利用联邦学习（FL）作为LLMs指令微调的学习框架。这是首次探索基于FL的LLMs指令微调方法。鉴于文本数据主要由终端用户生成，这一研究尤为重要。因此，必须设计和调整联邦学习方法，以有效利用存储于本地设备上的用户多样化指令，同时保护隐私并确保数据安全。本文通过广泛采用的GPT-4自动评估证明，利用客户端异构且多样化的指令集，结合所提出的FedIT框架，与仅依赖有限本地指令的集中式训练相比，我们显著提升了LLMs的性能。此外，我们开发了一个名为Shepherd的GitHub仓库，该仓库为探索跨不同类别异构指令的LLMs联邦微调提供了基础框架。