In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties -- like asymptotic unbiasedness and proper uncertainty quantification -- which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80--90\%. To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.

翻译：在计算社会科学（CSS）中，研究者通过分析文档来解释社会和政治现象。大多数情况下，CSS研究者首先获取文档标签，然后在第二步通过可解释的回归分析来解释这些标签。一种日益普及的低成本大规模文档标注方法是使用大语言模型（LLMs）。然而，与其他可扩展的标注生产方式类似，这种代理标签往往存在不完善和有偏的问题。我们提出了一种新算法，能够在保证统计性质（如渐近无偏性和恰当的不确定性量化）的同时，使用不完美的标注代理进行下游统计分析——这些性质是CSS研究的基石。研究表明，即使代理标签准确率高达80-90%，在下游统计分析中直接使用这些标签仍会导致显著偏差和无效置信区间。为解决这一问题，我们在去偏机器学习基础上提出设计式监督学习（DSL）估计量。DSL采用双稳健过程，将代理标签与少量高质量金标准标签相结合。通过控制文档被采样进行金标准标注的概率，我们的方法能保证下游统计分析的稳健推断，即使代理标签存在任意偏差且无需严苛假设。理论分析和实验结果表明，DSL在提供有效统计推断的同时，其均方根误差与仅关注预测而无推断保证的现有替代方法相当。