We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.

翻译：我们提出了大规模法律嵌入基准（MLEB），这是迄今为止规模最大、多样性最丰富、最全面的开源法律信息检索基准。MLEB包含十个由专家标注的数据集，涵盖多个司法管辖区（美国、英国、欧盟、澳大利亚、爱尔兰和新加坡）、多种文档类型（案例、立法、监管指南、合同和文献）以及多种任务类型（检索、零样本分类和问答）。为了填补开源法律信息检索领域中存在的领域和司法管辖区空白，MLEB中的七个数据集是全新构建的。我们详细记录了构建MLEB及创建其新组成数据集的方法论，并公开发布了我们的代码、结果和数据，以支持可复现的评估。