Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.

翻译：近年来，受Transformer架构惊人进展的推动，法律自然语言处理领域经历了显著增长。为了衡量进展，精心策划且具有挑战性的基准至关重要。然而，大多数基准仅限英语，尤其在法律NLP领域，目前尚无可用的多语言基准。此外，许多基准已经饱和，最佳模型明显超越人类顶尖水平，并达到近乎完美的分数。我们调研了法律NLP文献，选取了覆盖24种语言的11个数据集，构建了LEXTREME。为了提供公平比较，我们提出了两个综合分数：一个基于数据集，另一个基于语言。最佳基线模型（XLM-R large）在数据集综合分数和语言综合分数上均达到61.3分。这表明LEXTREME仍然非常具有挑战性，并留有充足的改进空间。为了方便研究人员和从业者使用，我们在Hugging Face上发布了LEXTREME，并附带了评估模型所需的所有代码，以及一个包含所有实验运行的公共Weights and Biases项目。