Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not necessarily emphasize addressing these concerns in their workflows. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.

翻译：科学家常通过元分析来综合评估某项干预措施对特定结果的影响，但若对不同研究背景（如研究设计差异）中可能存在的关键变化取平均值，则会威胁元分析估计的实用性和有效性。证据质量与可公度性的不确定性使元分析结果受到质疑，然而现有元分析软件工具的工作流程并未充分强调解决这些问题。我们提出MetaExplorer——一个与元分析专家通过迭代设计共同开发的元分析原型系统，该系统提供引导式流程，帮助用户获取不确定性评估结果并推理如何将其纳入统计推断过程。对经验丰富的元分析师进行的质性评估表明，结构化工作流程的引入既提升了认知性问题的感知重要性，也为工具创造契机，促使用户围绕证据聚合的目标与标准展开对话。