The mass aggregation of knowledge embedded in large language models (LLMs) holds the promise of new solutions to problems of observability and measurement in the social sciences. We examine the utility of one such model for a particularly difficult measurement task: measuring the latent ideology of lawmakers, which allows us to better understand functions that are core to democracy, such as how politics shape policy and how political actors represent their constituents. We scale the senators of the 116th United States Congress along the liberal-conservative spectrum by prompting ChatGPT to select the more liberal (or conservative) senator in pairwise comparisons. We show that the LLM produced stable answers across repeated iterations, did not hallucinate, and was not simply regurgitating information from a single source. This new scale strongly correlates with pre-existing liberal-conservative scales such as NOMINATE, but also differs in several important ways, such as correctly placing senators who vote against their party for far-left or far-right ideological reasons on the extreme ends. The scale also highly correlates with ideological measures based on campaign giving and political activists' perceptions of these senators. In addition to the potential for better-automated data collection and information retrieval, our results suggest LLMs are likely to open new avenues for measuring latent constructs like ideology that rely on aggregating large quantities of data from public sources.

翻译：大型语言模型（LLMs）中嵌入的大量知识聚合为社会科学中可观测性与测量问题提供了新的解决方案。我们考察了其中一种模型在特殊测量任务中的效用：量化立法者的潜在意识形态，这有助于更深入地理解民主核心功能，例如政治如何影响政策制定、政治人物如何代表选民。通过引导ChatGPT对第116届美国国会参议员进行成对比较（选择更倾向自由主义或保守主义的一方），我们沿自由-保守光谱对参议员进行了量化排序。研究表明，该LLM在重复迭代中输出稳定结果，未出现幻觉现象，且并非简单复述单一来源信息。这一新量表与既有自由-保守量表（如NOMINATE）高度相关，但在关键维度存在差异——例如能准确识别因极端左翼或右翼意识形态立场而偏离党派投票的参议员，并将其归入光谱极端位置。该量表还与基于竞选捐款和政治活动家对参议员认知的意识形态测量指标高度相关。除可能实现更优的自动化数据收集与信息检索外，我们的结果表明，LLM有望为测量依赖公共数据大规模聚合的潜在构念（如意识形态）开辟新路径。