We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

翻译：我们提出SeaEval，一个用于评估多语言基础模型的基准。除刻画模型理解与推理自然语言的能力外，我们还探究了它们对文化实践、文化细微差异及文化价值观的理解程度。除标准准确率指标外，我们进一步考察了基础模型在语义与多语言维度上的脆弱性。分析涵盖开源与闭源模型，获得了经典自然语言处理任务、推理及文化理解方面的实证结果。关键发现包括：(1) 多数模型在面对释义指令时表现各异；(2) 许多模型仍存在暴露偏差（如位置偏差、多数标签偏差）；(3) 对于植根于事实、科学及常识知识的问题，跨语义等价的多语言查询本应产生一致响应，但多数模型却表现出令人意外的不一致性；(4) 多语言训练的模型尚未具备"均衡多语言"能力。本研究强调亟需更具泛化性的语义表征与增强的多语言语境化能力。SeaEval可作为深入探究与评估多语言及多文化场景的跳板。