Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.

翻译：摘要：近年来，受Transformer架构显著进步的推动，法律自然语言处理领域取得了惊人的发展。为衡量进展，精心设计且具有挑战性的基准至关重要。然而，大多数基准仅针对英语，而在法律自然语言处理领域，目前尚无多语言基准可用。此外，许多基准已趋饱和，最优模型的性能明显超越顶尖人类专家，并几乎达到完美分数。我们调研了法律自然语言处理文献，遴选出覆盖24种语言的11个数据集，构建了LEXTREME。为提供公平比较，我们提出了两个聚合分数：一个基于数据集，另一个基于语言。最佳基线模型（XLM-R大型）在数据集聚合分数和语言聚合分数上均达到61.3分。这表明LEXTREME仍极具挑战性，存在充足的改进空间。为便于研究人员和从业者使用，我们已在Hugging Face上发布LEXTREME，并附带评估模型所需的全部代码，以及记录所有运行实验的公开Weights & Biases项目。