The evaluation of English text embeddings has transitioned from evaluating a handful of datasets to broad coverage across many tasks through benchmarks such as MTEB. However, this is not the case for multilingual text embeddings due to a lack of available benchmarks. To address this problem, we introduce the Scandinavian Embedding Benchmark (SEB). SEB is a comprehensive framework that enables text embedding evaluation for Scandinavian languages across 24 tasks, 10 subtasks, and 4 task categories. Building on SEB, we evaluate more than 26 models, uncovering significant performance disparities between public and commercial solutions not previously captured by MTEB. We open-source SEB and integrate it with MTEB, thus bridging the text embedding evaluation gap for Scandinavian languages.

翻译：英语文本嵌入的评估已从少数数据集的评估转变为通过MTEB等基准在众多任务上的广泛覆盖。然而，由于缺乏可用的基准，多语言文本嵌入的评估尚未实现这一转变。为解决此问题，我们引入了斯堪的纳维亚嵌入基准（SEB）。SEB是一个综合性框架，能够针对斯堪的纳维亚语言在24个任务、10个子任务和4个任务类别上进行文本嵌入评估。基于SEB，我们评估了超过26个模型，揭示了公开模型与商业解决方案之间显著的性能差异，这些差异此前未被MTEB所捕捉。我们将SEB开源并与MTEB集成，从而弥补了斯堪的纳维亚语言在文本嵌入评估方面的空白。