As large language models increase in capability, researchers have started to conduct surveys of all kinds on these models with varying scientific motivations. In this work, we examine what we can learn from language models' survey responses on the basis of the well-established American Community Survey (ACS) by the U.S. Census Bureau. Using a de-facto standard multiple-choice prompting technique and evaluating 40 different language models, hundreds of thousands of times each on questions from the ACS, we systematically establish two dominant patterns. First, models have significant position and labeling biases, for example, towards survey responses labeled with the letter "A". Second, when adjusting for labeling biases through randomized answer ordering, models across the board trend towards uniformly random survey responses. In fact, binary classifiers can almost perfectly differentiate between models' responses to the ACS and the responses of the US census. Taken together, our findings suggest caution in treating survey responses from language models as equivalent to those of human populations at present time.

翻译：随着大型语言模型能力不断提升，研究人员已开始基于各种科学动机对这些模型开展各类调查。本研究以美国人口普查局成熟的美国社区调查（ACS）为基础，探讨我们能从语言模型的调查回复中获得何种启示。通过采用事实上的标准多选提示技术，对40种不同语言模型分别进行数十万次ACS问题测试，我们系统性地发现两种主导模式：其一，模型存在显著的位置和标签偏差（例如对标注字母"A"的调查回复存在偏好）；其二，当通过随机化答案顺序调整标签偏差后，所有模型的调查回复均趋向于均匀随机分布。事实上，二元分类器几乎能完美区分模型对ACS的回复与美国人口普查的真实数据。综合来看，我们的发现表明：当前将语言模型的调查回复等同于人类群体回复的做法需保持谨慎。