Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not provide an explicit software representation of these concerns. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.

翻译：科学家常通过元分析来综合评估某项干预对某一结局指标的影响。然而，当研究者对不同研究设计等可能重要的背景差异取平均值时，元分析估计的效用和有效性将面临威胁。关于证据质量与可比性的不确定性对元分析结果产生质疑，但现有元分析软件工具并未提供这些问题的显式软件表示。我们提出了MetaExplorer——一个通过迭代设计与元分析专家共同开发的元分析原型系统，该系统通过引导式流程，促使研究者系统评估不确定性，并探讨如何在统计推断中纳入这些考量。对资深元分析专家的定性评估表明，结构化工作流的引入既提升了认知关切的重要性，也为工具引导用户围绕证据整合的目标与标准展开对话提供了契机。