Objective: Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation. Materials and Methods: The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations. Results: Our results show that, with an appropriate setup, these models can achieve state of the art performance. The best run, using few-shot learning, achieved 0.58 macro F1 score on publication abstracts and 0.75 macro F1 score on clinical observations, the former being comparable with the state of the art, while the latter surpassing the current best in class tool. Conclusion: While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.

翻译：目的：临床深度表型分析与表型注释在罕见病患者的诊断以及构建可计算处理的罕见病领域知识中均扮演关键角色。这些过程依赖于使用本体概念（通常来自人类表型本体）结合表型概念识别任务（通常由机器学习方法支持）来整理患者档案或现有科学文献。随着大型语言模型在多数自然语言处理任务中的广泛应用，我们评估了支撑ChatGPT的最新生成式预训练Transformer模型在临床表型分析与表型注释任务中的基础性能。材料与方法：实验设置包含七个不同特异性级别的提示、两个GPT模型（gpt-3.5-turbo和gpt-4.0）以及两个已建立的表型识别黄金标准语料库（分别包含论文摘要和临床观察记录）。结果：结果显示，在适当设置下，这些模型能达到业界最优性能。采用少样本学习的最佳实验在论文摘要上获得0.58宏F1分数，在临床观察记录上获得0.75宏F1分数——前者与当前最优水平相当，后者则超越了现有最佳工具。结论：尽管结果令人鼓舞，但结果的不确定性、高昂成本以及同一提示和输入在不同运行间缺乏一致性，使这些大型语言模型在该特定任务中的应用存在挑战。