As large language models increase in capability, researchers have started to conduct surveys of all kinds on these models with varying scientific motivations. In this work, we examine what we can learn from a model's survey responses on the basis of the well-established American Community Survey (ACS) by the U.S. Census Bureau. Evaluating more than a dozen different models, varying in size from a few hundred million to ten billion parameters, hundreds of thousands of times each on questions from the ACS, we systematically establish two dominant patterns. First, smaller models have a significant position and labeling bias, for example, towards survey responses labeled with the letter "A". This A-bias diminishes, albeit slowly, as model size increases. Second, when adjusting for this labeling bias through randomized answer ordering, models still do not trend toward US population statistics or those of any cognizable population. Rather, models across the board trend toward uniformly random aggregate statistics over survey responses. This pattern is robust to various different ways of prompting the model, including what is the de-facto standard. Our findings demonstrate that aggregate statistics of a language model's survey responses lack the signals found in human populations. This absence of statistical signal cautions about the use of survey responses from large language models at present time.

翻译：随着大型语言模型能力的提升，研究者们开始基于不同的科学动机对这些模型进行各类调查。本研究基于美国人口调查局成熟的美国社区调查项目，探讨我们能从模型调查回复中获取何种信息。我们评估了十余个参数量从数亿到百亿不等的不同模型，对每个模型就ACS问卷问题进行了数十万次测试，系统性地确立了两种主导模式。其一，较小模型存在显著的立场与标签偏差，例如倾向于选择标有字母"A"的调查选项。这种"A偏向"会随模型规模增大而缓慢减弱。其二，在通过随机化答案顺序调整标签偏差后，模型仍不会趋向于美国人口统计数据或任何可识别群体的统计特征。相反，所有模型均一致趋向于调查回复的均匀随机聚合统计量。这一模式在包括事实标准在内的多种提示方式下均保持稳定。我们的研究结果表明，语言模型调查回复的聚合统计量缺乏人类群体中存在的信号特征。这种统计信号的缺失对当前阶段使用大型语言模型的调查回复提出了警示。