Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments. Using N=76 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analysis with the original findings and human reanalysis. For 7 studies, the LLM could not produce a viable effect size estimate. For the remaining studies, our LLM pipeline recovered the original effect sizes in 41% of studies using a +/-0.05 tolerance in Cohen's d. Further, our LLM pipeline reached the same qualitative conclusion as the original study in 96% of cases, where conclusions indicate whether the reanalysis supports the original claim. For comparison, human reanalysts recovered the original effect sizes in 34% of studies and reached the same qualitative conclusion in 74% of cases. Together, these results show that LLMs can serve as a scalable tool for automated reproducibility assessment and provide a foundation for systematic auditing of empirical results in the social and behavioral sciences.

翻译：社会科学与行为科学中的可重复性通常由独立研究人员通过重新分析原始数据来评估，以判断已发表的研究结果能否复现。然而，这类方法资源消耗大且难以规模化推广。本研究表明，大语言模型（LLMs）能够自动化可重复性评估。我们选取n=76项已发表研究（这些研究包含行为与社会科学领域的预设论断），将LLM生成的分析结果与原始研究数据及人工复现结果进行对比。在7项研究中，LLM无法生成有效的效应量估计值。对于其余研究，我们的LLM流程在Cohen's d允许±0.05容差范围内，成功复现了41%研究的原始效应量。更关键的是，该流程在96%的案例中得出了与原始研究一致的定性结论（结论表明复现分析是否支持原始论断）。作为对照，人工复现团队在34%的研究中复现了原始效应量，并在74%的案例中得出相同定性结论。综合而言，这些结果表明LLM可作为自动化可重复性评估的可扩展工具，为社会科学与行为科学领域实证结果的系统性审计奠定了方法论基础。