Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.

翻译：文本嵌入模型的评估通常局限于少量任务，这些任务在语言、领域和任务多样性方面均存在约束。为突破这些限制并提供更全面的评估，我们引入了大规模多语言文本嵌入基准测试（MMTEB）——这是对MTEB的大规模社区驱动扩展，涵盖超过250种语言的500余项质量受控评估任务。MMTEB包含多样化的挑战性新型任务，例如指令跟随、长文档检索和代码检索，构成了迄今为止规模最大的多语言嵌入模型评估任务集合。基于该集合，我们开发了若干高度多语言的基准测试，并用于评估代表性模型集合。研究发现，虽然拥有数十亿参数的大语言模型（LLMs）能在特定语言子集和任务类别上达到最先进性能，但当前公开可用的最佳模型是仅含5.6亿参数的multilingual-e5-large-instruct。为提升可访问性并降低计算成本，我们提出基于任务间相关性的新型下采样方法，在保持模型相对排序的同时确保任务选择的多样性。此外，我们通过采样困难负例对检索等任务进行优化，创建了规模更小但效能相当的子集。这些优化使我们能够推出显著降低计算需求的基准测试。例如，我们新推出的零样本英语基准测试在保持与完整版本相似排序顺序的同时，仅需极低比例的计算成本。