The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) $\rightarrow$ answer" from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely $11.1\%$, $25.0\%$, $72.2\%$, and $75.0\%$ of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve $<0.2$ Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.

翻译：对多元文化人类价值观的感知能力，是语言模型生成安全且个性化回答的关键。然而，由于计算机科学领域缺乏关于多元文化价值观的大规模真实世界数据，语言模型的这种感知能力尚未得到充分研究。本文提出WorldValuesBench——一个面向多元文化价值预测任务的全球多样性大规模基准数据集，该任务要求模型基于人口统计背景对价值问题生成评分响应。我们的数据集源自具有影响力的社会科学项目"世界价值观调查"，该项目收集了全球94,728名参与者对数百个价值问题（如社会、经济、伦理问题）的答案。基于WVS数据，我们构建了超过2000万个形如"(人口统计属性，价值问题)→答案"的样本。通过案例研究，我们发现该任务对强大的开源和闭源模型均具有挑战性：在全部问题中，Alpaca-7B、Vicuna-7B-v1.5、Mixtral-8x7B-Instruct-v0.1和GPT-3.5 Turbo分别仅能在11.1%、25.0%、72.2%和75.0%的问题上实现与人类归一化答案分布的Wasserstein-1距离小于0.2。WorldValuesBench为研究语言模型多元文化价值感知的局限性及机遇开辟了新的研究方向。