We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

翻译：我们提出SeaEval，一个用于评估多语言基础模型的基准。除刻画这些模型理解及推理自然语言的能力外，本基准还探究其对文化习俗、细微差异与价值观的理解程度。除标准准确率指标外，我们进一步从语义与多语言性维度剖析基础模型的脆弱性。分析涵盖开源与闭源模型，在经典自然语言处理任务、推理及文化理解任务上形成实证结论。核心发现包括：(1) 多数模型面对经改写表达指令时表现各异；(2) 许多模型仍存在暴露偏差（如位置偏差、多数标签偏差）；(3) 对于基于事实、科学及常识知识的问题，模型对语义等价的多语言查询应给出一致响应，然而多数模型惊人的表现相悖；(4) 经多语言训练的模型尚未实现“均衡的多语言能力”。我们的工作凸显了提升语义表征泛化性及强化多语言语境化的必要性。SeaEval可作为深入探究与评估多语言、多文化场景的起点。