Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We evaluate eleven open-weight LLMs (7-123 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3.3-70B-Instruct is the highest performing model, achieving the best results on 8/16 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 11 tasks, we also evaluate three GPT-4 and GPT-4o series models and find comparable results to Llama-3.3-70B-Instruct. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions.

翻译：大语言模型（LLMs）的进展引发了人们对其在公共卫生等多个领域辅助人类专家潜力的浓厚兴趣。本研究针对涉及自由文本分类与提取的公共卫生任务，对大语言模型进行了自动化评估。我们整合了六个外部标注数据集和七个新内部标注数据集，以评估大语言模型在处理以下主题文本时的表现：健康负担、流行病学风险因素和公共卫生干预措施。我们采用零样本上下文学习方法，在全部任务上评估了十一个开放权重的大语言模型（参数量70亿至1230亿）。研究发现，Llama-3.3-70B-Instruct是性能最优的模型，在8/16的任务中取得了最佳结果（基于微平均F1分数）。不同任务间表现差异显著：所有开放权重大语言模型在某些挑战性任务（如接触分类）上的微平均F1分数均低于60%，而在其他任务（如胃肠道疾病分类）上所有模型均能获得超过80%的微平均F1分数。在11项任务的子集上，我们还评估了三个GPT-4和GPT-4o系列模型，发现其表现与Llama-3.3-70B-Instruct相当。总体而言，基于这些初步结果，我们认为大语言模型有望成为公共卫生专家从多样化的自由文本源中提取信息、支持公共卫生监测、研究和干预的有用工具。