Commonsense knowledge, a major constituent of artificial intelligence (AI), is primarily evaluated in practice by human-prescribed ground-truth labels. An important, albeit implicit, assumption of these labels is that they accurately capture what any human would think, effectively treating human common sense as homogeneous. However, recent empirical work has shown that humans vary enormously in what they consider commonsensical; thus what appears self-evident to one benchmark designer may not be so to another. Here, we propose a method for assessing commonsense knowledge in AI, specifically in large language models (LLMs), that incorporates empirically observed heterogeneity among humans by measuring the correspondence between a model's judgment and that of a human population. We first find that, when treated as independent survey respondents, most LLMs remain below the human median in their individual commonsense competence. Second, when used as simulators of a hypothetical population, LLMs correlate with real humans only modestly in the extent to which they agree on the same set of statements. In both cases, smaller, open-weight models are surprisingly more competitive than larger, proprietary frontier models. Our evaluation framework, which ties commonsense knowledge to its cultural basis, contributes to the growing call for adapting AI models to human collectivities that possess different, often incompatible, social stocks of knowledge.

翻译：常识知识作为人工智能（AI）的重要组成部分，在实践中主要通过人工预设的真实标签进行评估。这些标签隐含着一个重要假设：它们能准确捕捉任何人类的认知，从而将人类常识视为同质的。然而，最近的实证研究表明，人类对于何为常识存在巨大差异；因此，对一位基准设计者而言不言自明的事实，对另一位可能并非如此。本文提出一种评估AI（特别是大型语言模型（LLMs））中常识知识的方法，该方法通过衡量模型判断与人类群体判断之间的一致性，纳入实证观察到的人类异质性。我们首先发现，当被视为独立的调查受访者时，大多数LLMs在个体常识能力上仍低于人类中位数水平。其次，当用作假设群体的模拟器时，LLMs在相同陈述集上的认同程度与真实人类的相关性仅处于中等水平。在这两种情况下，较小型的开源权重模型的表现出人意料地优于较大型的专有前沿模型。我们的评估框架将常识知识与其文化基础相联系，响应了日益增长的呼吁：使AI模型适应那些拥有不同且往往互不相容的社会知识储备的人类群体。