The aggregation of knowledge embedded in large language models (LLMs) holds the promise of new solutions to problems of observability and measurement in the social sciences. We examine this potential in a challenging setting: measuring latent ideology -- crucial for better understanding core political functions such as democratic representation. We scale pairwise liberal-conservative comparisons between members of the 116th U.S. Senate using prompts made to ChatGPT. Our measure strongly correlates with widely used liberal-conservative scales such as DW-NOMINATE. Our scale also has interpretative advantages, such as not placing senators who vote against their party for ideologically extreme reasons towards the middle. Our measure is more strongly associated with political activists' perceptions of senators than other measures, consistent with LLMs synthesizing vast amounts of politically relevant data from internet/book corpora rather than memorizing existing measures. LLMs will likely open new avenues for measuring latent constructs utilizing modeled information from massive text corpora.

翻译：大型语言模型（LLM）中嵌入的知识聚合，有望为社会科学的可观测性与测量问题提供新的解决方案。我们在一个具有挑战性的设定中考察这一潜力：测量潜在意识形态——这对于更深入理解民主代表性等核心政治功能至关重要。我们利用对ChatGPT的提示，对第116届美国参议院议员进行成对的自由-保守比较。我们的测量结果与广泛使用的自由-保守量表（如DW-NOMINATE）高度相关。该量表同时具备解释优势，例如不会将因意识形态极端原因而反对本党的参议员归入中间派。与政治活动家对参议员看法的关联性方面，我们的测量结果优于其他测量方式，这符合LLM从互联网/书籍语料库中综合大量政治相关数据（而非记忆现有测量结果）的特性。LLM有望利用大规模文本语料库中的建模信息，为测量潜在构念开辟新路径。