Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not provide an explicit software representation of these concerns. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.

翻译：科学家常通过元分析来评估某项干预措施对特定结果的影响，并汇总多篇文献中的证据。然而，当研究者将不同研究设计等可能重要的情境差异进行平均化时，元分析估计的实用性和有效性会受到威胁。证据质量与可比性的不确定性对元分析结果的可信度产生质疑，但现有元分析软件工具并未提供明确表示这些问题的软件机制。我们提出MetaExplorer——一个采用迭代设计方法、与元分析专家共同开发的原型系统，该系统通过引导式流程来启发研究者对不确定性进行评估，并推理如何将其纳入统计推断过程。我们对有经验的元分析专家进行的定性评估表明，强制性的结构化工作流程不仅提升了认知问题的重要性感知，还为用户提供了工具介入围绕证据整合目标与标准展开对话的契机。