Meta-analyses statistically aggregate the findings of different randomized controlled trials (RCTs) to assess treatment effectiveness. Because this yields robust estimates of treatment effectiveness, results from meta-analyses are considered the strongest form of evidence. However, rigorous evidence syntheses are time-consuming and labor-intensive, requiring manual extraction of data from individual trials to be synthesized. Ideally, language technologies would permit fully automatic meta-analysis, on demand. This requires accurately extracting numerical results from individual trials, which has been beyond the capabilities of natural language processing (NLP) models to date. In this work, we evaluate whether modern large language models (LLMs) can reliably perform this task. We annotate (and release) a modest but granular evaluation dataset of clinical trial reports with numerical findings attached to interventions, comparators, and outcomes. Using this dataset, we evaluate the performance of seven LLMs applied zero-shot for the task of conditionally extracting numerical findings from trial reports. We find that massive LLMs that can accommodate lengthy inputs are tantalizingly close to realizing fully automatic meta-analysis, especially for dichotomous (binary) outcomes (e.g., mortality). However, LLMs -- including ones trained on biomedical texts -- perform poorly when the outcome measures are complex and tallying the results requires inference. This work charts a path toward fully automatic meta-analysis of RCTs via LLMs, while also highlighting the limitations of existing models for this aim.

翻译：荟萃分析通过统计汇总不同随机对照试验（RCTs）的发现来评估治疗效果。由于这种方法能产生稳健的治疗效果估计，荟萃分析的结果被视为最强形式的证据。然而，严格的证据综合过程既耗时又费力，需要手动从单个试验中提取数据以供综合。理想情况下，语言技术应能按需实现全自动的荟萃分析。这需要从单个试验中准确提取数值结果，而目前自然语言处理（NLP）模型尚不具备这一能力。在本研究中，我们评估了现代大型语言模型（LLMs）能否可靠地执行此任务。我们标注（并发布）了一个适度但细粒度的临床试验报告评估数据集，其中包含与干预措施、对照和结局相关的数值结果。利用该数据集，我们评估了七种LLM在零样本条件下条件性提取试验报告数值结果的表现。我们发现，能够处理长文本输入的大规模LLM已接近实现全自动荟萃分析，尤其适用于二分类结局（如死亡率）。然而，对于结局指标复杂且需要推断才能汇总结果的情况，LLM（包括在生物医学文本上训练的模型）表现不佳。本研究为通过LLM实现RCT全自动荟萃分析指明了方向，同时也指出了现有模型在此目标上的局限性。