Reproducibility remains a central challenge in computational social science, where complex workflows, evolving software ecosystems, and inconsistent documentation hinder researchers ability to re-execute published methods. This study presents a systematic evaluation of reproducibility across three conditions: uncurated documentation, curated documentation, and curated documentation paired with a preset execution environment. Using 47 usability test sessions, we combine behavioral performance indicators (success rates, task time, and error profiles) with questionnaire data and thematic analysis to identify technical and conceptual barriers to reproducibility. Curated documentation substantially reduced repository-level errors and improved users ability to interpret method outputs. Standardizing the execution environment further improved reproducibility, yielding the highest success rate and shortest task completion times. Across conditions, participants frequently relied on AI tools for troubleshooting, often enabling independent resolution of issues without facilitator intervention. Our findings demonstrate that reproducibility barriers are multi-layered and require coordinated improvements in documentation quality, environment stability, and conceptual clarity. We discuss implications for the design of reproducibility platforms and infrastructure in computational social science.

翻译：可复现性仍是计算社会科学领域的核心挑战，复杂的工作流程、不断演化的软件生态系统以及不一致的文档记录阻碍了研究者对已发表方法的重新执行。本研究通过三种条件系统评估可复现性：未经整理的文档、经过整理的文档，以及整理文档与预设执行环境的组合。通过47次可用性测试，我们结合行为绩效指标（成功率、任务时间与错误类型）、问卷数据及主题分析，识别出影响可复现性的技术性与概念性障碍。经过整理的文档显著减少了仓库层面的错误，并提升了用户解读方法输出的能力。标准化执行环境进一步改善了可复现性，实现了最高的成功率和最短的任务完成时间。在所有实验条件下，参与者频繁依赖AI工具进行问题排查，通常能在无需协助者干预的情况下独立解决问题。我们的研究结果表明，可复现性障碍具有多层次性，需要在文档质量、环境稳定性和概念清晰度方面进行协同改进。本文进一步讨论了其对计算社会科学领域可复现性平台与基础设施设计的启示。