Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in few- and zero-shot settings. However, it is unclear if such models are similarly capable in more specialized, high-stakes domains such as biomedicine. In this paper, we enlist domain experts (individuals with medical training) to evaluate summaries of biomedical articles generated by GPT-3, given zero supervision. We consider both single- and multi-document settings. In the former, GPT-3 is tasked with generating regular and plain-language summaries of articles describing randomized controlled trials; in the latter, we assess the degree to which GPT-3 is able to \emph{synthesize} evidence reported across a collection of articles. We design an annotation scheme for evaluating model outputs, with an emphasis on assessing the factual accuracy of generated summaries. We find that while GPT-3 is able to summarize and simplify single biomedical articles faithfully, it struggles to provide accurate aggregations of findings over multiple documents. We release all data and annotations used in this work.

翻译：大型语言模型，特别是GPT-3，能够在少样本和零样本设置下生成通用领域新闻文章的高质量摘要。然而，此类模型在生物医学等专业性更强、风险更高的领域是否具有类似能力尚不清楚。本文邀请领域专家（接受过医学培训的人员）评估GPT-3在零监督条件下生成的生物医学文献摘要。我们考虑了单文档和多文档两种场景：前者中，GPT-3需为描述随机对照试验的文章生成常规摘要和通俗语言摘要；后者中，我们评估GPT-3综合多篇文献所报告证据的能力。我们设计了用于评估模型输出的标注方案，重点考察生成摘要的事实准确性。研究发现，尽管GPT-3能够忠实地总结和简化单篇生物医学文献，但在跨多篇文档准确汇总研究结果方面仍存在困难。本研究公开了所有使用的数据与标注结果。