Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability.

翻译：可重复性与可复制性面临的挑战已引起广泛关注，这得益于成功率不甚理想的大规模重复研究项目。目前已有初步研究开发出用于评估已发表研究发现可复制性的算法。本研究探讨如何将基于人工智能的研究置信度信号整合到文献检索中。我们访谈了17位博士研究员，了解其当前的文献检索流程，并征求其对可复制性评估工具的反馈意见。研究结果表明，参与者容易将可复制性与可推广性及相关概念混淆。可复制性信息可在研究设计全过程中为研究者提供支持。然而，由于缺乏可解释性和透明度，人工智能评估的适用性仍存争议。此类工具若要获得广泛接受，必须进一步研究人工智能辅助置信度评估的伦理影响。我们讨论了相关技术工具设计的启示，以支持学术活动并推动可复制性发展。