Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We initially evaluate five open-weight LLMs (7-70 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3-70B-Instruct is the highest performing model, achieving the best results on 15/17 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 12 tasks, we also evaluate GPT-4 and find comparable results to Llama-3-70B-Instruct, which scores equally or outperforms GPT-4 on 6 of the 12 tasks. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions.

翻译：大型语言模型（LLMs）的进展引发了人们对其在公共卫生等多个领域辅助人类专家潜力的广泛关注。本研究针对涉及自由文本分类与提取的公共卫生任务，对LLMs进行了自动化评估。我们整合了六个外部标注数据集与七个新构建的内部标注数据集，以评估LLMs在处理以下主题文本时的表现：健康负担、流行病学风险因素及公共卫生干预措施。我们首先采用零样本上下文学习方法，在全部任务上评估了五个开放权重的LLMs（参数量70亿至700亿）。研究发现，Llama-3-70B-Instruct是性能最优的模型，在15/17的任务中取得了最佳结果（依据微平均F1分数）。不同任务间表现差异显著：所有开放权重LLMs在部分挑战性任务（如接触途径分类）上的微平均F1分数均低于60%，而在其他任务（如胃肠道疾病分类）上所有LLMs均能获得超过80%的微平均F1分数。在12项任务的子集上，我们还评估了GPT-4模型，发现其与Llama-3-70B-Instruct表现相当；在12项任务中的6项上，Llama-3-70B-Instruct与GPT-4持平或更优。总体而言，基于这些初步结果，我们认为LLMs展现出成为公共卫生领域实用工具的潜力，能够帮助专家从多样化的自由文本源中提取信息，并为公共卫生监测、研究及干预措施提供支持。