Challenges to reproducibility and replicability have gained widespread attention over the past decade, driven by a number of large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate, or predict, the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a prototype replicability estimation tool. Our findings suggest that information about replicability can support researchers throughout literature review and research design processes. However, explainability and interpretability of system outputs is critical, and potential drawbacks of AI-enabled confidence assessment need to be further studied before such tools could be widely accepted and deployed. We discuss implications for the design of technological tools to support scholarly activities and advance reproducibility and replicability.

翻译：过去十年间，因多个大型重复性项目成功率表现平平，可再现性与可重复性面临的挑战已获得广泛关注。新兴研究已开始开发算法以估算或预测已发表研究成果的可重复性。本研究探索将人工智能驱动的研究可信度信号整合到文献检索中的可行路径。我们通过访谈17位博士研究人员，了解其现有文献检索流程，并收集他们对可重复性估测原型工具的反馈。研究结果表明：关于可重复性的信息能够在文献综述与科研设计全流程中为研究者提供支持。然而，系统输出的可解释性与可理解性至关重要，且AI驱动可信度评估的潜在缺陷需经进一步研究，此类工具方能获得广泛接受与部署。我们讨论了技术工具设计对学术活动的支撑作用以及提升可再现性与可重复性的启示。