Introduction: Recent work suggests large language models (LLMs) can accelerate screening, but prior evaluations focus on earlier LLMs, standardized Cochrane reviews, single-model setups, and accuracy as the primary metric, leaving generalizability, configuration effects, and calibration largely unexamined. Methods: We developed OLIVER (Optimized LLM-based Inclusion and Vetting Engine for Reviews), an open-source pipeline for LLM-assisted abstract screening. We evaluated multiple contemporary LLMs across two non-Cochrane systematic reviews and performance was assessed at both the full-text screening and final inclusion stages using accuracy, AUC, and calibration metrics. We further tested an actor-critic screening framework combining two lightweight models under three aggregation rules. Results: Across individual models, performance varied widely. In the smaller Review 1 (821 abstracts, 63 final includes), several models achieved high sensitivity for final includes but at the cost of substantial false positives and poor calibration. In the larger Review 2 (7741 abstracts, 71 final includes), most models were highly specific but struggled to recover true includes, with prompt design influencing recall. Calibration was consistently weak across single-model configurations despite high overall accuracy. Actor-critic screening improved discrimination and markedly reduced calibration error in both reviews, yielding higher AUCs. Discussion: LLMs may eventually accelerate abstract screening, but single-model performance is highly sensitive to review characteristics, prompting, and calibration is limited. An actor-critic framework improves classification quality and confidence reliability while remaining computationally efficient, enabling large-scale screening at low cost.

翻译：引言：近期研究表明大型语言模型（LLM）可加速文献筛选，但现有评估多聚焦早期LLM模型、标准化Cochrane综述、单模型配置及以准确率为核心指标，其泛化性、配置效应与校准性能尚未得到充分检验。方法：我们开发了开源流水线OLIVER（面向综述的优化LLM纳入与审核引擎），用于LLM辅助摘要筛选。通过在两项非Cochrane系统综述中测试多种当代LLM模型，采用准确率、AUC及校准指标在全文筛选和最终纳入阶段评估性能。进一步构建了融合两个轻量化模型的actor-critic筛选框架，并在三种聚合规则下进行测试。结果：各独立模型表现差异显著。在规模较小的综述1（821篇摘要，63篇最终纳入）中，多个模型对最终纳入文献展现出高灵敏度，但伴随大量假阳性及较差校准性能。在规模较大的综述2（7741篇摘要，71篇最终纳入）中，多数模型特异性较高却难以有效识别真阳性文献，提示词设计显著影响召回率。单模型配置虽整体准确率高，但校准性能普遍薄弱。Actor-critic筛选框架在两项综述中均提升了判别能力，显著降低校准误差，并获得更高AUC值。讨论：LLM有望加速摘要筛选进程，但单模型性能对综述特征与提示词设计高度敏感，且校准能力有限。Actor-critic框架在保持计算效率的同时，提升了分类质量与置信度可靠性，为低成本大规模筛选提供了可行路径。