When designing evidence-based policies and programs, decision-makers must distill key information from a vast and rapidly growing literature base. Identifying relevant literature from raw search results is time and resource intensive, and is often done by manual screening. In this study, we develop an AI agent based on a bidirectional encoder representations from transformers (BERT) model and incorporate it into a human team designing an evidence synthesis product for global development. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. To further improve team efficiency, we enhance the human-AI hybrid team through active learning (AL). Specifically, we explore different sampling strategies, including random sampling, least confidence (LC) sampling, and highest priority (HP) sampling, to study their influence on the collaborative screening process. Results show that incorporating the BERT-based AI agent into the human team can reduce the human screening effort by 68.5% compared to the case of no AI assistance and by 16.8% compared to the case of using a support vector machine (SVM)-based AI agent for identifying 80% of all relevant documents. When we apply the HP sampling strategy for AL, the human screening effort can be reduced even more: by 78.3% for identifying 80% of all relevant documents compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming workflow in the design process of three evidence gap maps (EGMs) for USAID and find it to be highly effective. These findings demonstrate how AI can accelerate the development of evidence synthesis products and promote timely evidence-based decision making in global development in a human-AI hybrid teaming context.

翻译：在制定基于证据的政策和项目时，决策者必须从海量且快速增长的基础文献中提炼关键信息。从原始搜索结果中识别相关文献耗时且资源密集，通常需通过人工筛选完成。本研究基于双向编码器表示转换模型（BERT）开发了一个AI智能体，并将其整合到设计全球发展证据综合产品的人类团队中。我们探究了人机混合团队在加速证据综合过程中的有效性。为进一步提升团队效率，我们通过主动学习（AL）优化人机混合团队。具体而言，我们探索了随机采样、最低置信度（LC）采样和最高优先级（HP）采样等不同采样策略，以研究其对协作筛选过程的影响。结果表明，在识别80%相关文献时，将基于BERT的AI智能体纳入人类团队后，人工筛选工作量较无AI辅助场景减少68.5%，较使用基于支持向量机（SVM）的AI智能体场景减少16.8%。当采用HP采样策略进行主动学习时，人工筛选工作量可进一步降低：在识别80%相关文献时较无AI辅助减少78.3%。我们将AL增强的人机混合团队工作流程应用于美国国际开发署（USAID）的三份证据缺口图（EGM）设计过程，发现其具有高效率。这些发现揭示了在人机混合团队情境下，AI如何加速证据综合产品的开发，并促进全球发展中基于证据的及时决策。