In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties -- like asymptotic unbiasedness and proper uncertainty quantification -- which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90%. To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.

翻译：在计算社会科学（CSS）中，研究者通过分析文档来解释社会与政治现象。通常情况下，CSS研究者首先获取文档标签，随后在第二阶段利用可解释的回归分析解释这些标签。大规模低成本标注文档的一种日益普遍的方法是通过大型语言模型（LLMs）。然而，与其他可扩展的标注生成方式类似，这类替代标签往往存在不完善性和偏差。本文提出了一种新算法，利用不完美的替代标注进行下游统计分析，同时保证CSS研究核心的统计性质——如渐近无偏性和合理的不确定性量化。研究表明，即使替代标签的准确率高达80-90%，直接将其用于下游统计分析仍会导致显著偏差和无效置信区间。为解决这一问题，我们基于去偏机器学习技术，提出了设计型监督学习（DSL）估计量。DSL采用双重稳健方法，将替代标签与少量高质量金标准标签相结合。通过控制文档被抽中进行金标准标注的概率，我们的方法能够保证下游统计分析的推断有效性，即使替代变量存在任意偏差且无需严格假设。理论分析与实验结果表明，DSL在提供有效统计推断的同时，其均方根误差与仅关注预测而无推断保证的现有替代方法相当。