We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

翻译：我们提出SeaEval，一个用于多语言基础模型的评估基准。除刻画这些模型理解与推理自然语言的能力外，我们还探究其如何把握文化实践、细微差异及价值观念。在标准准确率指标之外，我们进一步调查基础模型在语义与多语言维度上的脆弱性。我们的分析涵盖开源与闭源模型，并在经典自然语言处理任务、推理能力及文化理解层面获得实证结果。关键发现表明：（1）面对改写后的指令，多数模型会呈现不一致的行为；（2）许多模型仍存在曝光偏差（如位置偏差、多数标签偏差）；（3）对于根植于事实、科学及常识知识的问题，语义等价的多语言查询理应获得一致回答，但令人惊讶的是，多数模型在此类查询上表现不一致；（4）经过多语言训练的模型尚未实现"均衡多语言"能力。我们的工作凸显了构建更可泛化的语义表征与增强多语言语境化的必要性。SeaEval可作为一个平台，支撑针对多语言与多文化场景的更深入调查与评估。