Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in few- and zero-shot settings. However, it is unclear if such models are similarly capable in more specialized, high-stakes domains such as biomedicine. In this paper, we enlist domain experts (individuals with medical training) to evaluate summaries of biomedical articles generated by GPT-3, given zero supervision. We consider both single- and multi-document settings. In the former, GPT-3 is tasked with generating regular and plain-language summaries of articles describing randomized controlled trials; in the latter, we assess the degree to which GPT-3 is able to \emph{synthesize} evidence reported across a collection of articles. We design an annotation scheme for evaluating model outputs, with an emphasis on assessing the factual accuracy of generated summaries. We find that while GPT-3 is able to summarize and simplify single biomedical articles faithfully, it struggles to provide accurate aggregations of findings over multiple documents. We release all data and annotations used in this work.

翻译：大型语言模型，尤其是GPT-3，能够在零样本和少样本场景下生成高质量的通用领域新闻文章摘要。但这类模型在生物医学等更专业、高风险领域是否同样适用尚不明确。本文邀请领域专家（具有医学训练背景的人员）在零监督条件下评估GPT-3生成的生物医学文章摘要。我们考虑了单文档和多文档两种场景：前者的任务是让GPT-3对描述随机对照试验的文章生成常规摘要和简明语言摘要；后者则评估GPT-3综合多篇文献中报告证据的能力。我们设计了一套标注方案用于评估模型输出，重点关注生成摘要的事实准确性。研究发现，GPT-3虽能忠实地总结和简化单篇生物医学文章，但在准确整合多篇文献的发现方面存在困难。本研究公开了所有使用的数据与标注。