Federated learning has enabled multiple parties to collaboratively train large language models without directly sharing their data (FedLLM). Following this training paradigm, the community has put massive efforts from diverse aspects including framework, performance, and privacy. However, an unpleasant fact is that there are currently no realistic datasets and benchmarks for FedLLM and previous works all rely on artificially constructed datasets, failing to capture properties in real-world scenarios. Addressing this, we propose FedLLM-Bench, which involves 8 training methods, 4 training datasets, and 6 evaluation metrics, to offer a comprehensive testbed for the FedLLM community. FedLLM-Bench encompasses three datasets (e.g., user-annotated multilingual dataset) for federated instruction tuning and one dataset (e.g., user-annotated preference dataset) for federated preference alignment, whose scale of client number ranges from 38 to 747. Our datasets incorporate several representative diversities: language, quality, quantity, instruction, length, embedding, and preference, capturing properties in real-world scenarios. Based on FedLLM-Bench, we conduct experiments on all datasets to benchmark existing FL methods and provide empirical insights (e.g., multilingual collaboration). We believe that our FedLLM-Bench can benefit the FedLLM community by reducing required efforts, providing a practical testbed, and promoting fair comparisons. Code and datasets are available at https://github.com/rui-ye/FedLLM-Bench.

翻译：联邦学习使得多方能够在不直接共享数据的情况下协作训练大语言模型（FedLLM）。遵循这一训练范式，研究社区从框架、性能与隐私等多个方面投入了巨大努力。然而，一个令人遗憾的事实是，目前尚无针对FedLLM的真实数据集与基准测试，先前工作均依赖人工构建的数据集，未能捕捉真实场景中的特性。为此，我们提出FedLLM-Bench，其包含8种训练方法、4个训练数据集与6项评估指标，旨在为FedLLM社区提供一个全面的测试平台。FedLLM-Bench涵盖三个用于联邦指令微调的数据集（例如用户标注的多语言数据集）以及一个用于联邦偏好对齐的数据集（例如用户标注的偏好数据集），其客户端数量规模从38到747不等。我们的数据集整合了多种代表性差异维度：语言、质量、数量、指令、长度、嵌入表示与偏好，从而捕捉真实场景中的特性。基于FedLLM-Bench，我们在全部数据集上进行了实验，以对现有联邦学习方法进行基准评估，并提供实证分析（例如多语言协作）。我们相信，FedLLM-Bench可通过降低所需投入、提供实用测试平台及促进公平比较，使FedLLM社区受益。代码与数据集发布于https://github.com/rui-ye/FedLLM-Bench。