Large Language Models (LLMs) have gained prominence in the field of Legal Intelligence, offering potential applications in assisting legal professionals and laymen. However, the centralized training of these Legal LLMs raises data privacy concerns, as legal data is distributed among various institutions containing sensitive individual information. This paper addresses this challenge by exploring the integration of Legal LLMs with Federated Learning (FL) methodologies. By employing FL, Legal LLMs can be fine-tuned locally on devices or clients, and their parameters are aggregated and distributed on a central server, ensuring data privacy without directly sharing raw data. However, computation and communication overheads hinder the full fine-tuning of LLMs under the FL setting. Moreover, the distribution shift of legal data reduces the effectiveness of FL methods. To this end, in this paper, we propose the first Federated Legal Large Language Model (FedJudge) framework, which fine-tunes Legal LLMs efficiently and effectively. Specifically, FedJudge utilizes parameter-efficient fine-tuning methods to update only a few additional parameters during the FL training. Besides, we explore the continual learning methods to preserve the global model's important parameters when training local clients to mitigate the problem of data shifts. Extensive experimental results on three real-world datasets clearly validate the effectiveness of FedJudge. Code is released at https://github.com/yuelinan/FedJudge.

翻译：[translated abstract in Chinese] 大语言模型（LLMs）在法律智能领域日益受到重视，为辅助法律专业人士及普通民众提供了潜在应用前景。然而，这些法律LLMs的集中式训练引发了数据隐私问题，因为法律数据分布在不同机构中，且包含敏感的个人信息。本文通过探索法律LLMs与联邦学习方法（FL）的融合来应对这一挑战。采用FL后，法律LLMs可在设备或客户端本地进行微调，其参数通过中央服务器聚合与分发，从而在无需直接共享原始数据的情况下确保数据隐私。然而，计算与通信开销阻碍了联邦学习场景下LLMs的完全微调。此外，法律数据的分布偏移降低了FL方法的有效性。为此，本文首次提出联邦法律大语言模型（FedJudge）框架，能够高效且有效地微调法律LLMs。具体而言，FedJudge采用参数高效的微调方法，在FL训练过程中仅更新少量附加参数。同时，我们探索了持续学习方法，在训练本地客户端时保留全局模型的重要参数，以缓解数据偏移问题。在三个真实数据集上的大量实验结果清晰验证了FedJudge的有效性。代码已开源至https://github.com/yuelinan/FedJudge。