The mass aggregation of knowledge embedded in large language models (LLMs) holds the promise of new solutions to problems of observability and measurement in the social sciences. We examine the utility of one such model for a particularly difficult measurement task: measuring the latent ideology of lawmakers, which allows us to better understand functions that are core to democracy, such as how politics shape policy and how political actors represent their constituents. We scale the senators of the 116th United States Congress along the liberal-conservative spectrum by prompting ChatGPT to select the more liberal (or conservative) senator in pairwise comparisons. We show that the LLM produced stable answers across repeated iterations, did not hallucinate, and was not simply regurgitating information from a single source. This new scale strongly correlates with pre-existing liberal-conservative scales such as NOMINATE, but also differs in several important ways, such as correctly placing senators who vote against their party for far-left or far-right ideological reasons on the extreme ends. The scale also highly correlates with ideological measures based on campaign giving and political activists' perceptions of these senators. In addition to the potential for better-automated data collection and information retrieval, our results suggest LLMs are likely to open new avenues for measuring latent constructs like ideology that rely on aggregating large quantities of data from public sources.

翻译：大规模语言模型（LLMs）中嵌入的知识聚合，有望为社会科学中的可观测性和测量问题提供新解决方案。我们考察了此类模型在一项尤为困难的测量任务中的效用：测量立法者的潜在意识形态。这使我们能够更深入理解民主的核心功能，例如政治如何塑造政策，以及政治行动者如何代表其选民。我们通过提示ChatGPT进行成对比较，选择更倾向自由主义（或保守主义）的参议员，对第116届美国国会参议员进行自由主义-保守主义光谱的标度。研究表明，LLM在重复迭代中产生了稳定答案，未出现幻觉现象，且并非简单复述单源信息。这一新标度与NOMINATE等现有自由主义-保守主义标度高度相关，但在若干重要方面存在差异，例如能正确地将因极左或极右意识形态原因而背离本党投票的参议员置于极端位置。该标度还与基于竞选捐款和政治活动家对议员认知的意识形态测量高度相关。除实现更优自动化数据收集与信息检索的潜力外，我们的结果表明，LLM有望为依赖聚合海量公开数据来测量意识形态等潜在构念开辟新路径。