Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, there are few datasets available for specialized critical domains such as law and the available ones are often only for the English language. We curate and release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. The MultiLegalPile corpus, which includes diverse legal data sources with varying licenses, allows for pretraining NLP models under fair use, with more permissive licenses for the Eurlex Resources and Legal mC4 subsets. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, the trained models, and all of the code under the most open possible licenses.

翻译：大规模高质量的数据集对于训练大型语言模型至关重要。然而，目前专门针对法律等关键领域的可用数据集很少，且现有数据集往往仅涵盖英语。我们整理并发布了MultiLegalPile，这是一个包含来自17个司法管辖区的24种语言的689GB语料库。该语料库整合了多种许可协议下的多样化法律数据源，允许在合理使用原则下预训练自然语言处理模型，其中Eurlex资源与Legal mC4子集采用更为宽松的许可协议。我们基于该语料库预训练了两个RoBERTa模型、一个多语言Longformer模型，以及针对各语言子集的24个单语模型，并在LEXTREME基准上进行了评估。此外，我们还在LexGLUE基准上评估了英语模型与多语言模型的表现。我们的多语言模型在LEXTREME上取得了新的最优结果，英语模型则在LexGLUE上实现了突破。数据集、训练模型及相关代码均以尽可能开放的许可协议发布。