AI science evaluation tools aim to assess research credibility. As with traditional metrics such as impact factors, their edicts can be decontextualised and repurposed in problematic ways. To address this, I propose Critically-Engaged Pragmatism as a scientific norm enjoining scientific communities to scrutinise the purposes and purpose-specific reliability of AI science evaluation tools. To foster Critically Engaged Pragmatism, creators of AI science evaluation tools should transparently and fully report design, training, and benchmarking details to facilitate assessments of purpose-specific reliability, liability to different types of error, and bias. What count as best practices for the transparent reporting of AI science evaluation tools should be updated as new forms of error, bias, and gamesmanship are discovered. Under this framework, AI science evaluation tools are not objective arbiters of scientific credibility. Rather, they are the object of critical discursive practices that ultimately ground the credibility of scientific communities.

翻译：AI科学评估工具旨在评估研究的可信度。与传统指标（如影响因子）类似，其指令可能脱离语境并以有问题的方式被重新利用。为解决这一问题，我提出批判性参与实用主义作为一种科学规范，要求科学界审视AI科学评估工具的目的及其针对特定目的的可靠性。为促进批判性参与实用主义，AI科学评估工具的创造者应透明且全面地报告设计、训练和基准测试细节，以便评估工具针对特定目的的可靠性、对不同类型错误和偏见的责任。随着新形式的错误、偏见和操纵手段被发现，AI科学评估工具透明报告的最佳实践应随之更新。在此框架下，AI科学评估工具并非科学可信度的客观仲裁者。相反，它们是批判性话语实践的对象，而这些实践最终构成了科学界可信度的基础。