We hypothesize that large language models (LLMs) based on the transformer architecture can enable automated detection of clinical phenotype terms, including terms not documented in the HPO. In this study, we developed two types of models: PhenoBCBERT, a BERT-based model, utilizing Bio+Clinical BERT as its pre-trained model, and PhenoGPT, a GPT-based model that can be initialized from diverse GPT models, including open-source versions such as GPT-J, Falcon, and LLaMA, as well as closed-source versions such as GPT-3 and GPT-3.5. We compared our methods with PhenoTagger, a recently developed HPO recognition tool that combines rule-based and deep learning methods. We found that our methods can extract more phenotype concepts, including novel ones not characterized by HPO. We also performed case studies on biomedical literature to illustrate how new phenotype information can be recognized and extracted. We compared current BERT-based versus GPT-based models for phenotype tagging, in multiple aspects including model architecture, memory usage, speed, accuracy, and privacy protection. We also discussed the addition of a negation step and an HPO normalization layer to the transformer models for improved HPO term tagging. In conclusion, PhenoBCBERT and PhenoGPT enable the automated discovery of phenotype terms from clinical notes and biomedical literature, facilitating automated downstream tasks to derive new biological insights on human diseases.

翻译：我们假设基于Transformer架构的大型语言模型（LLMs）能够实现临床表型术语（包括HPO中未记录的术语）的自动检测。在本研究中，我们开发了两种模型：PhenoBCBERT（基于BERT的模型，采用Bio+Clinical BERT作为预训练模型）和PhenoGPT（基于GPT的模型，可从多种GPT模型初始化，包括GPT-J、Falcon和LLaMA等开源版本，以及GPT-3和GPT-3.5等闭源版本）。我们将提出的方法与近期开发的结合规则与深度学习方法的HPO识别工具PhenoTagger进行了比较。结果表明，我们的方法能提取更多表型概念（包括HPO未收录的新概念）。我们还通过生物医学文献案例研究，展示了如何识别和提取新的表型信息。我们从模型架构、内存占用、运行速度、准确性和隐私保护等多个维度，对比了基于BERT和基于GPT的表型标注模型。同时，我们探讨了在Transformer模型中增加否定步骤和HPO归一化层以改进HPO术语标注的方法。结论表明，PhenoBCBERT和PhenoGPT能够从临床记录和生物医学文献中自动发现表型术语，为后续通过自动化下游任务获取人类疾病的新生物学见解提供了支持。